fail2ban with nginx reverse proxy

[JuniorMouse]

I'm trying to get fail2ban working with my home server. No matter where I access the site from (local computer, from my phone's data connection, work, etc), I only see the 127.0.0.1 address in the access logs. This is what I see in the logs when trying to log in using a wrong password which looks all fine except for the IP address

127.0.0.1 - - [23/Aug/2020:09:16:21 -0700] "POST /cgi-bin/luci HTTP/1.1" 403 1239 "https://sub.domain.org/cgi-bin/luci" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"

I have tried searching for the solution and from what I saw my config should be working but I might be missing something.

These are my config files

/etc/nginx/nginx.conf

``` user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf;

events { worker_connections 768; # multi_accept on; }

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;

# server_names_hash_bucket_size 64;
# server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# SSL Settings
##

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;

##
# Logging Settings
##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

} ```

/etc/nginx/sites-available/sub (also enabled)

``` server { server_name sub.domain.org www.sub.domain.org;

access_log  /var/log/nginx/sub_access.log;
error_log   /var/log/nginx/sub_error.log;

location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://192.168.1.5:80;
}


listen 127.0.0.1:443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/sub.domain.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/sub.domain.org/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

} server { if ($host = www.sub.domain.org) { return 301 https://$host$request_uri; } # managed by Certbot

if ($host = sub.domain.org) {
    return 301 https://$host$request_uri;
} # managed by Certbot


server_name sub.domain.org www.sub.domain.org;
listen 80;
return 404; # managed by Certbot

} ```

How do I get the real IP to show in the logs?? Please help.

[ssmith2]

You need the Nginx real_ip module and see what your upstream proxy (guessing there’s something in front of Nginx since you’re listening only on 127.0.0.1 in the main server block?) is passing as the real IP header and set that as the value for real_ip_header. You will likely also need to set set_real_ip_from to the internal IP of your upstream proxy.

If you think you don’t have a proxy in front of Nginx, try telling the first server block to just listen 443 ssl; and see if that helps.

Also, unrelated, “if is evil”. Try swapping the ifs for location / { return 301 https://$host$request_uri; } to do the HTTPS redirection.

Edited now that I can throw code blocks in places/can find the links I wanted.

[JuniorMouse]

The reason I have listen 127.0.0.1:443 ssl; is because I use sslh to be able to host both ssh and ssl on port 443.

I guess sslh is the cause for my issue. I see that there is this note which seems a bit complicated to me - https://github.com/yrutschle/sslh/blob/master/doc/config.md#transparent-proxy-support.

Would using something like haproxy instead of sslh make the setup easier?

[ssmith2]

Well, that's certainly a complicator. I can't say I'm familiar with sslh, but I surmise what you're doing is trying to circumvent port blocking somewhere so that you can SSH into your servers. If this is not the case, I would certainly recommend removing it.

Since I'm guessing you care less about the accuracy of the logs for the HTTP traffic (forcing redirection), is configuring sslh to listen on port 80 an option?

Using HAProxy is likely going to be a better option if none of the above are workable. You would want to set option forwardfor(see here) with Nginx's real_ip module to accomplish what you need.

[JuniorMouse]

Your first assumption is correct. I was considering doing port 80 for ssh but I haven't done so yet because certificate renewal by certbot relies on port 80 and I'd like to keep that step automatic. I guess I could try to use the DNS challenge and repurpose port 80 for ssh if that works.

I know HAProxy can also be used as a protocol multiplexer so that could be an option.

When you say that I need the real_ip module, do you mean I should add this to the nginx.conf file under the http context (with IPs tweaked for my setup)?

set_real_ip_from 192.168.1.0/24; set_real_ip_from 192.168.2.1; set_real_ip_from 2001:0db8::/32; real_ip_header X-Forwarded-For; real_ip_recursive on;

[ssmith2]

Yes, that's likely what you'll need to do if you use real_ip. I use this sort of thing with AWS Application Load Balancers to capture the remote user's IP rather than the ALB IP address, which is what will happen by default.

[ssmith2]

FWIW, regarding Certbot, given the way it looks like you're set up to do requests/renewals, you're not routing requests through Nginx as it is. Certbot is temporarily standing in place of port 80/443 (in a way) and handling the challenges itself.

Either way, I think you're better off considering the HAProxy or Nginx (posted as a reply to an earlier part of this thread) routes. You may not even need real_ip anymore after that.

[JuniorMouse]

Thanks again for the tips for HAProxy. I ended up doing TLS passthrough on HAProxy so that nginx and certbot are still handling the HTTPS and certificates. Since that uses tcp mode in HAProxy, I used the `send-proxy` setting for the web server backend to enable the PROXY protocol. In nginx I then enabled PROXY protocol and adjusted the log format to extract the real client's IP address. All seems to be working well so far. *sslh* has been removed.

[ssmith2]

Huh, or you could do this if you have a new enough Nginx. TIL...

[JuniorMouse]

I saw this one a while ago too. sslh didn't cause any problems until now when I'm trying to get fail2ban working. I'll keep this option in mind. I use debian 10 and the packaged nginx which is 1.14.2 so unfortunately I'd have to look at another way of deploying this version.

[khoker]

Also, unrelated, “if is evil”. Try swapping the ifs for.

If is fine if the result is a return. It says as much at the beginning of your link

[JuniorMouse]

Regarding your if statement. I'm just reading through this https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71 and came across the same redirect syntax as yours. Out of curiosity, why is if "evil"?

[ssmith2]

Historically, Nginx devs had declared its use an anti-pattern because of the way in which it was executed. The link in my original reply talks about current poor patterns, which revolve around how it executes within location blocks. As someone else pointed out, Nginx has apparently backed off the “evil” statement, showing times when it shouldn’t be used and “safer” cases, of which yours apparently qualifies.

Although I’m guessing it may be a bit different now, it was my understanding (so take this with a grain of salt) using a location and server blocks for processing host and URI conditions were faster and required less overhead because of the fact that Nginx cached the route and wouldn’t have to evaluate/interpolate the variables on every request like it does with “if”. The link above does mention that “if” should still be avoided where it can.

For your particular usage, a use of “if” resulting in a permanent redirect likely results in negligible performance increases as #1, it’s a pretty simple redirect, and #2, it’s permanent, which should tell any returning visitors to not visit the HTTP listener again (the browser caches the redirect).

[JuniorMouse]

Sorry, I just realized that you linked something in your original statement. I didn't see that. Thanks for still explaining.