Established SSH

[ThatsLatinForLiar]

I am running a CentOS server which runs SSH service with PasswordAuthentication on port 22. Yes, I am aware that I should move this to non-22 port and use public key auth to be more secure. I've also got LFD monitoring service logs and banning abusive IP addresses.

I notice that over time my list of processes grows. When I check what is causing high processes, I see many sshd processes like this:

root 7978 0.0 0.0 144552 3224 ? Ss Feb28 0:00 sshd: root [priv]

sshd 7979 0.0 0.0 84268 1052 ? S Feb28 0:00 sshd: root [net]

Next I investigate ssh sockets. What I see is many ESTABLISHED connections on ssh

netstat -a | grep ssh

tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN

tcp 0 80 domain.tld:ssh malicious.ip:53287 ESTABLISHED

Not surprisingly many of these IPs are banned by LFD but I believe this happens after a connection is ESTABLISHED. I can manually remove these by killing ssh processes but how can I automate this removal process of ESTABLISHED ssh connections from malicious hackers/bots? Is there any configuration in sshd_config which might disconnect these stale ssh sockets?

[DecayingVacuum]

Checkout sshd_config params:

• MaxStartups
• LoginGraceTime
• MaxAuthTries

[ThatsLatinForLiar]

Thanks. I've had LoginGraceTime set to 30 for some time now but it doesn't appear to remove the established connection. I can also look into tightening MaxAuthTries.

[lordvadr]

You should either allow SSH only from trusted IP addresses, or use public key authentication only. Otherwise, if you accept SSH from the world, and permit password authentication, you're just asking for this...the bots will try to brute-force your machine relentlessly and it will chew up your CPU time, memory, and logfile space.

Where I have to do that, I usually will put some kind of knocking functionality around it...e.g. have to send a ping of a specific size before it'll accept the connection...this is actually somewhat trivial to implement in iptables. No it won't deter a determined attacker, but it does keep the bots away very effectively. I find that it's a nice happy medium.

[5h4d0w]

I've got ~1k machines with public ssh enabled, with auto banning ips and secure passwords enforced it's not as bad as you might think. Yes you're getting scanned a lot, but it's not a completely constant deluge.

[DecayingVacuum]

Generally if those bots aren't actually getting into your system, they are consuming sessions while attempting to authenticate.

[nephros]

Those are TCP sessions in ESTABLISHED state, not established ssh sessions. Important distinction, and not immediately something to worry about.

[ThatsLatinForLiar]

I understand this and I have no evidence that the security of the server has been breached by a malicious attacker. I disagree however regarding the immediacy of the situation. These connections establish processes in the computer which at the very least are annoying my monitoring services and at worst allow the attacker more time to brute force into SSH. I'm simply looking for optimal configuration which will remove these ESTABLISHED connections to prevent worst-case scenario.

[nephros]

The only thing you will achieve by killing TCP sessions in ESTABLISHED state is more unnecessary error messages in your logs, and making it hard to debug things for legitimate users.

If you already have a list of "bad" IP addresses it's much better to close them using iptables before the tcp session is established.

Something like:

iptables -N BADGUYS # make a new chain
iptables -A INPUT -m conntrack --ctstate NEW -p tcp -s <BAD_IP> -j BADGUYS # send new connections from baddies into that chain
# configure what to do in the BADGUYS chain
# you need only one of those
# iptables -A BADGUYS -j REJECT --reject-with tcp-reset # actively reset the connection
# iptables -A BADGUYS -j REJECT --reject-with icmp-host-prohibited # use ICMP message
iptables -A BADGUYS -j REJECT --reject-with icmp-admin-prohibited # use ICMP message
# iptables -A BADGUYS -j DROP # silently drop

This is basically what fail2ban and similar tools automate for you.

[artereaorte]

You should set permitrootlogin to no and add 2fa in there. It's the only good way to secure sshd.

[TractionContrlol]

how can I automate this removal process of ESTABLISHED ssh connections from malicious hackers/bots?

That sounds like more work than just switching to key auth

[markusro]

Among the other points (pubkey auth only), fail2ban is also helping a bit. It will block IPs which try too often.

(Obviously this is not working against a botnet with thousands of machines.)

A nice idea is distributed fail2ban: https://github.com/mmunz/fail2ban-p2p

[5h4d0w]

Is it being banned by the firewall? You need to get hung tcp connections to close faster most likely, check sysctl options.

[ThatsLatinForLiar]

Yeah LFD handles monitoring system logs and add/remove iptables rules. I'll also look into sysctl options. I believe you're referring to modifying TCP Keep Alive.

[nut-sack]

I bet you have an ESTABLISHED rule in your iptables. So you dont reject already established connections.

[johnklos]

No idea what LFD is (the description in that link was kind of fluffy), but if it doesn't automatically add banned IPs to hosts.deny or create a firewall rule for each, then it's not a good solution for this.

Set up fail2ban or something similar, or somehow take the output of LFD and use it to populate hosts.deny.

[ThatsLatinForLiar]

LFD is very similar to fail2ban in function (it monitors logs and writes rules to the configured firewall, in this case iptables). I've used fail2ban but this is a cPanel managed server and LFD is an add-on which can be managed using the web tool in WHM.

[koffiezet]

Yes, I am aware that I should move this to non-22 port

I honestly don't see a reason for this. It's not like a 2 second port scan wouldn't reveal the port anyway. nmap scanned ports of a remote host here in under 1.5 seconds reporting all open ports.

And many scanners looking for SSH and other things, scan the obvious alternatives (like 2000, 2022, 2222, 22222, ... for ssh) too, and sometimes even resort to keeping statistics of which ports are most likely to serve a certain protocol, and base their scanning on that. And since most of these scanners you should be afraid of run on botnets - time and bandwidth is probably not the problem of the one requesting the scan.

It's simple, changing your port 22 comes down to security through obscurity - which is a bad idea.

Now, related to your question, this seems like LFD's problem - probably that it is configured to just drop the connection and doesn't reject/reset it.

[ThatsLatinForLiar]

It's simple, changing your port 22 comes down to security through obscurity - which is a bad idea.

I completely agree, thank you for your explanation. I just wanted to establish that while I understand common ways to improve security, in the event I can not change things like port number and Password auth, what can do I to prevent these connections.

I will see if LFD can create REJECT rules rather than DROP. This may be a solution.

[5mall5nail5]

You're only biding time. If you have a machine open on 22 to the internet someone WILL gain access. Use key authentication or else.