How to deny ip address automatically by parsing nginx log file

Question

I have 4 webservers behind cloudflare and a loadbalancer, nginx is the webserver, php-fpm manages the php pages. I don't know how to block a simple dos attack ...

I'm able to detect this attack by using the http_limit_req module from nginx http://wiki.nginx.org/HttpLimitReqModule

but this is not blocking the attack at all, yes this can mitigate but webservers are hit and hit again, and php-fpm goes to 80% and in a minute the website is unreachable.

I'm trying to find a way to block this kind of request.

I know how to block certain ip address or certain useragent with nginx but i want to do it automatically. I think that I cannot block the ip with iptables because the request come from the loadbalancer :( but i'm still able to detect the correct ip address with the set_real_ip_from and real_ip_header X-Forwarded-For with nginx.

I have the log file (error.log) filled with the correct ip address as you can see:

2012/03/27 18:34:02 [error] 31234#0: *1283 limiting connections by zone "staging", client: XX.XX.XX.XXX, server: www.xxxxxxx.com, request: "HEAD /it HTTP/1.1", host: "www.xxxxxxx.com"

Someone have an idea and can teach me how to block automatically this ip?

Which part are you struggling with? Parsing the file? Adding IPs to any kind of blocklist? — Nico Haase
– Nico Haase, Commented Oct 22, 2024 at 9:19

luukes · Accepted Answer · 2012-03-28 07:40:52Z

6

use fail2ban for this. It's a log-file parser for many different services which can detect failed logins, etc. and then block an IP-address.

http://www.fail2ban.org

Regards

answered Mar 28, 2012 at 7:40

luukes

1,6001 gold badge10 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ilmetu Over a year ago

yes i think this is a good solutions, i'm trying to setup fail2ban with this hint : forum.nginx.org/read.php?11,223078 i will let you know :) thanks

cgiacomi Over a year ago

Is it possible to do the same with CSF instead of Fail2Ban?

lubosdz · Accepted Answer · 2024-10-22 09:17:34Z

There couple of techniques to mitigate DDoS attacks in nginx:

limiting the number of connections
limiting the rate of requests
closing down slow connections
forbidding access to certain directories
allowing or denying IP/IP range or user agent
setting up caching on nginx side or in your app

Take a look at blog Mitigating DDoS attacks.

Some other examples:

# forbid empty header
if ($http_user_agent = "") {
    return 403; 
}

# forbid various agents & crawlers containing forbidden keyword
if ($http_user_agent ~* (java|resty|python|http-client|httpClient|curl|TestBot) ) {
    return 403; 
}

# deny/allow IP access
location / {
    deny 5.39.218.201; # deny single IP
    deny 45.146.166.0/24; # deny range 45.146.166.0 - 45.146.166.255
    #allow 10.20.30.40; # allow single IP
    #allow 10.20.30.0/24; # allow range 10.20.30.0 - 10.20.30.255
}

# deny access to specific directories
location ~ ^/(protected|framework|themes/\w+/views|wp-|feed|xmlrpc) {
    deny all;
}

To test specific headers use ie. curl or wget:

> curl -I -L -k -H "User-agent: BadBot" http://tested.site
# flags: -H = header, -L = follow redirects http->https, -k = dont verify SSL cert

> wget --user-agent "BadBot" http://127.0.0.1

Collectives™ on Stack Overflow

How to deny ip address automatically by parsing nginx log file

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related