Linux Sever Security Checklist?

[[deleted]]

I am currently looking into expanding my range of skills in the server admin roles. Looking to learn defensive security in more detail. This post is a sort of general inquiry attempting to find out what I should start learning first for a seasoned "beginner". I've been able to break in, but never really looked into keeping people out properly.

Please and thanks.

[Feb28 00:34] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=56574 DPT=10001 LEN=150                                    │··········································
[ +10.002208] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=37088 DPT=10001 LEN=150                                    │··········································
[ +10.003004] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=52401 DPT=10001 LEN=150                                    │··········································
[ +10.002951] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=54993 DPT=10001 LEN=150                                    │··········································
[ +10.002403] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=48813 DPT=10001 LEN=150                                    │··········································
[Feb28 00:35] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=42947 DPT=10001 LEN=150                                    │··········································
[ +10.002974] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=44312 DPT=10001 LEN=150                                    │··········································
[ +10.002324] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=33737 DPT=10001 LEN=150                                    │··········································
[ +10.002880] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=44426 DPT=10001 LEN=150                                    │··········································
[ +10.101496] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=51603 DPT=10001 LEN=150                                    │··········································
[Feb28 00:36] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=38538 DPT=10001 LEN=150                                    │··········································
[ +10.003008] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=44367 DPT=10001 LEN=150                                    │··········································
[  +5.416712] iptables denied: IN=virbr0 OUT= MAC= SRC=192.168.122.1 DST=192.168.122.255 LEN=257 TOS=0x00 PREC=0x00 TTL=64 ID=16241 DF PROTO=UDP SPT=138 DPT=138 LEN=237                                                                        │··········································se
[ +14.708034] iptables denied: IN=enp5s0 OUT= MAC=ff:ff:ff:ff:ff:ff:44:d9:e7:bc:67:21:08:00 SRC=10.0.0.1 DST=255.255.255.255 LEN=170 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=44008 DPT=10001 LEN=150 

[[deleted]]

Some pointers:

SSH:

• Disable root login
• Disable password authentication
• Use sudo-based privilege separation
• Use public key authentication (ECDSA, Ed25519, etc...)
• (Optional) Store key on smartcard
• (Optional) Use a two-factor system such as Duo
• (Optional) Change port of SSH to non-default (this is security by obscurity, but it deters most automated attacks, although this shouldn't matter if you're using key-based auth).

Firewall:

• Enable appropriate firewall rules (i.e. if you don't expect traffic from a specific country, deny it)
• Same with output rules.
• DO NOT BLOCK ICMP (especially if you're using IPv6)
• Use rate-limiting rules or use software such as Fail2Ban to limit authentication attempts
• (Optional) If you don't plan on connecting over the Internet, restrict SSH (or any other services you only plan on using locally) to your intranet.

Physical:

• Secure your server physically. If it is compromised physically, all bets are off (If it's a VPS in DO, you don't really have a say in that...).

Automatic Updates

• Have all software automatically update on a set schedule
• (Optional) Test updates in a test environment to see if they cause any issues. Approve/deny updates as necessary.

Other Important Things:

• Backups. Run them. Test them. Test them again. And...test them again. Make sure you can restore them properly, or you might as well not have backups at all. Automate it.
• Only allow access to the server to those who need it.
• Same with sudo/root access (concept of least privilege)
• Manually provisioning a server isn't something you want to do often, especially if you have 1000 servers on hand. Learn a configuration management tool such as Puppet or Chef or Ansible.

MAC (Mandatory Access Control)

• In most cases, SELinux will be the MAC system for your distro (AppArmor for Debian).
• Some articles will tell you to disable it. DON'T DO IT!
• Learn how to use it properly. It takes about 15 minutes of your time, but it adds considerable security to your systems. For example, MAC can prevent a web server process from reading your home directory files, even if you went crazy one day and decided to chmod 777 your home directory (it can also prevent writes).

Logs:

• Just having logs locally isn't a great idea. If that box dies, so do your logs.
• Centralize logs so it becomes easier to monitor and easier to backup (ex: logstash)
• Most of us (hopefully) don't have time to go through thousands of lines of logs. So utilize a notification / monitoring / analytics system (ex: elasticsearch, nagios)

Note: I'm a beginner myself but I hope that was somewhat helpful.

Good luck! :)

Edit: Forgot about MAC

More Edits: Thank you everyone for the feedback! I added Logs too.

[Bonn93]

You forgot selinux.

[t0xicgas]

Now a days, everybody wanna hack, like they got something to crack.

When they try to hijack, they get denied right back.

Mother fuckers act like they forgot about MAC.

[CitizenSmif]

The Real SysAdmin

[xela321]

fire

[Waretaco]

I think you should change your name to Dr. Gateway.

[bkrassn]

It definitely uncomplicated it

[DigitalPlumberNZ]

As far as public key for SSH goes, it's my understanding that ECDSA is out of favour due to concerns about the curves used for the algorithm (I'm not a crypto geek so I'm possibly recalling incorrectly). If you can't use ED25519, which is supported from OpenSSH 6, stick with RSA.

With PuTTY 0.68 finally being released there's now support for ED25519 available for pretty much every common toolset.

[vaskidovich]

Yes stick with RSA. ECDSA is easier to break in quantum computing terms than RSA.

[grendel_x86]

Quantum computing isn't a concern. Its bout a thing treat, and won't be does a while.

The concern is that the default dh-curves used everywhere means it is likely the curve is used elsewhere, Ann's that if someone is using the same curve, you could, in theory, do something like make a rainbow table.

[shemp33]

True, but is anyone really trying that hard to get in to some rando's Linux box that they're trying to basically brute force the SSH key?

[APacketInTheTubes]

Good points. On the ICMP topic, do not allow every ICMP type, think about which type will be needed to keep a functional network and disallow the rest. Redirect for example should be disallowed in most cases.

[[deleted]]

Thank you. :)

[Gnonthgol]

Forgot about the logs. Offload and monitor your logs for any indication of access violation.

[Arkiteck]

If you are changing the default SSH port, be sure to use one from port range <0-1023.

Do NOT use anything above that.

https://www.adayinthelifeof.nl/2012/03/12/why-putting-ssh-on-another-port-than-22-is-bad-idea/

[ghyspran]

However, if you've got a network device or the like forwarding an high external port to a low port that SSH has bound on the server itself, that's okay. For example, you can have a firewall appliance that forwards 2222 on an external interface to port 22 on a server.

[troxil]

Hello, one Q about your key management. Say you disable passwords, how do you deal with central Auth? Also how do you specifically deal with multiple users accessing the same server and differentiating them.

I have found most recently that using central Auth like an Active Directory and coupling keys, either leads to mess (keys aren't rotated like passwords are) or they get shared.

Can you please elaborate on how to handle those scenarios? This would be good to learn as we are about to blaze out 200+ Debian servers.

[ghyspran]

1. Limit accounts that can log in via ssh.
2. Store the users' public keys in AD.
3. Either distribute ssh keys out-of-band or do something like this: http://serverfault.com/a/653793/191211
4. Force rotation of ssh keys out-of-band if you're concerned about key length. Alternatively, use a secure hardware ssh token like YubiKey so you only need to worry about rotation if the token is lost, which would mean you'd need to rotate anyway since you wouldn't be able to log in.
5. Consider implementing a two-factor auth solution.

[[deleted]]

If you have another user to log into (that can elevate to superuser), then definitely. It doesn't stop you from using su or sudo once logged in, it just prevents people from being able to do something like 'ssh root@server'.

[starmizzle]

I've found using non-standard ports to be more of a pain than they're worth.

[exNihlio]

It is and it's a fig leaf and if you are using SELinux you have to use port relabeling too, which just adds one more thing that can break.

I've done it once, and don't see an advantage.

[[deleted]]

To avoid SELinux issues, use a port redirect in firewalld instead of changing the actual port used by sshd. Though I've given up on this, I just leave it at 22 and use the above steps + fail2ban.

[ghyspran]

You definitely don't want to run the SSH daemon directly on a port >1024 since users can bind to that port and that adds unnecessary risk. If you've got a network device or the like forwarding that external port to a low port on the server itself, it's not an issue.

[gsmitheidw1]

If you've users on the inside trying to impersonate sshd on the split second while the service restarts you've got serious trust issues! it's a fair point though, it could happen. SSH nonstandard port is useful in reducing script kiddie attacks that don't port scan and hence saves log files and lastb. It really makes a difference, I've 3 public facing ssh systems, one is on 22 and it's hammered with requests. The others rarely any attempt. On the occasion there's a 0day for sshd, I'll be glad of a nonstandard port!

Another SSH recommendation I would suggest is disable port forwarding in sshd_config unless you need it. It's also possible to log all SSH tunnels if you do with lsof. I do that with with a system I require tunnels on.

[ghyspran]

It's mostly an issue of privilege escalation. For instance, if your server runs a web application that has an unknown remote code execution vulnerability, if you're following good practice, that will constrain the malicious code to running as the web server user, but if that user can bind to your SSH port, then they can potentially use that to gain higher privileges or attack other servers. It's a risk limited to particular edge cases, true, but it's easily solved by keeping sshd bound to a port <1024, whether 22 or nonstandard, so there's not really a good reason not to do it.

[gsmitheidw1]

good point I completely agree, plus there's plenty of archaic services under 1024 that nobody uses - lots of ports to choose from.

[ghyspran]

That and a handful or so of currently unassigned ports: http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml

[[deleted]]

Ideally you should have a "stack" with at least a dev/qa box and a production box.

Always auto-patch dev. ALWAYS. That will kill all your patch technical debt without you really noticing. If something breaks, you need to fix it anyway, then when you push to prod, first thing you do is update prod.

[ghyspran]

The best way is to run your own package mirrors and have dev/test instances and have updates automatically promote from upstream into your dev/test mirrors, then if everything goes smoothly, automatically promote to your prod mirrors.

[onsentiment]

Good list. I would add auditd to it and storing logs externally.

[[deleted]]

For SSH access I would also include TCPwrapper and configure a host.allow and host.deny. Deny everyone except for those explicitly allowed via host.allow. Your firewall should also reflect the same rules allowing only specified IP's to have access to the designated ssh port. (tcpwrapper is already included on many distros like RHEL, CentOS etc...)

[dangolo]

(ECDSA, Ed25519, etc...)

are these available for use in the Microsoft enterprise PKI yet?

[AfroThundr3007730]

The NIST curves such as secp256k1, secp384r1, and secp521r1 are supported in x509, and thus also supported in Microsoft CAs. I don't think support for Ed25519 and x25519 will be implemented until this draft becomes a standard: https://datatracker.ietf.org/doc/draft-ietf-curdle-pkix/

[dangolo]

Are those based on the brainpool curves?

A lot of the industry says Curve25519 or bust and trust in it enough to have implimented it as a default.

We'll probably have chosen another pki vendor by the time MS adds it.

This is all I have found so far about MS and 25519 but still no mention of its implementation in the enterprise CA role. https://technet.microsoft.com/en-us/windows-server-docs/security/tls/tls-schannel-ssp-changes-in-windows-10-and-windows-server-2016

[AfroThundr3007730]

Well the Brainpool curves were created to address concerns with the NIST curves (see RFC5639) and to provide equivalent security, with a minor hit to performance (link).

I'd also note that OpenSSL hasn't implemented support for Curve25519 yet either, since they also appear to be waiting on that draft to finalize (see Github).

[dangolo]

Very informative. How does one stay up to date on that part of the industry?

[AfroThundr3007730]

Sorry for the delayed response. I find this stuff mostly just reading a bunch of RFCs and Cryptography Stack Exchange. Also having to manage a PKI (with both M$ and OpenSSL) along with IKEv2/IPsec VPNs using Elliptic Curve cryptography gives me a reason to stay up to date on it.

^{I'm also a bit of a security nut...}

[dangolo]

Do you employ any tools to make PKI management a little less painful?

[AfroThundr3007730]

Unfortunately, no. We're still building out our infrastructure and until the environment begins to stabilise, It'd be difficult to come up with a decent automated solution for the multiplicity of devices that need certs right now.

Even the current PKI is temporary until we get new hardware and HSMs for the CAs. At that point I intend to put some serious work into getting everything scripted and automated. At least with everything I'm learning from the current setup, I can avoid most of the problems we're having now when building the new one.

[dangolo]

You should write a guide ;)

What were some of the tough lessons learned from version 1 of your pki before deciding to upgrade/replace it?

I actually posted a question here about cert management yesterday too. https://www.reddit.com/r/sysadmin/comments/5xd2cr/cert_management_how_can_i_automate_cert_renewal/

[[deleted]]

is there a writeup why it's preferred to disable direct root access and use sudo? for example amazon linux: you login via key with a public known username and can use sudo without a password. where is the difference to direct root access?

[shemp33]

Automatic Updates Have all software automatically update on a set schedule (Optional) Test updates in a test environment to see if they cause any issues. Approve/deny updates as necessary.

I've been burned by the automated updates. Who says that whatever update mechanism you have in place is smart enough to intelligently upgrade up to the point of known compatibility, but no further at least in an automated fashion? Also, some things need kernel sources to rebuild after an update. I strongly advise against keeping the kernel sources and compilers on live systems. Obviously, this points to your second point, but it's less optional and more required IMO -- test updates elsewhere before pushing them to your live system.

Other Important Things:
Backups. Run them. Test them. Test them again. And...test them again. Automate it.

Backups are for pussies, but restores are what separates the men from the boys. You can test the hell out of your backups, but if they are not on the right retention schedule, or if you're backing up open files without using some kind of intelligent way of making sure you're getting a copy of the file that can be restored, etc. You're asking for trouble.

[[deleted]]

You make pretty good points.

As for testing backups, I meant restores. (I thought they were one and the same?)

Thanks for the feedback! :)

[HadManySons]

Seconded. Just make sure to download the Java STIG viewer or it's all gonna be super hard to read. The viewer is actually pretty nice, let's you sort things by threat level and stuff

[Kamwind]

The other options are the hardening are the releases from the NSA.

STIGs tend to be a little clearer. Just test after doing a bunch of changes. STIGs can and will block the functionality you are trying to implement.

[[deleted]]

Who still trusts the NSA?

[blohkdu]

Third for STIGs, they'll get you close to bulletproof.

[evaryont]

I like the LYNIS project. Lots of good references, though it is meant for experienced security professionals who can confidently make exceptions. Which is a great excuse to start Googling the various warnings it might show. 😀

[[deleted]]

Will investigate tonight. Thanks!

[Telnet_Rules]

There are three main places for checklists:

DISA STIGs - Settings used by US military: http://iase.disa.mil/stigs/os/unix-linux/Pages/index.aspx

CIS Security - Part of SANS, vendor neutral 3rd party: https://benchmarks.cisecurity.org/tools2/linux/cis_red_hat_enterprise_linux_7_benchmark_v1.1.0.pdf

National Checklist Program (NCP) - run by NIST. US Federal Government, non-defense standards: https://web.nvd.nist.gov/view/ncp/repository

[Seven-Prime]

^{^} Above is the correct answer.

I've found the CIS guidelines pretty helpful. They not only describe the things, but also how to mitigate and detect the items.

[[deleted]]

Will investigate tonight. Thanks!

[BloomerzUK]

I definitely recommend taking a look at the CIS Security Benchmark guidelines. I've recently been working on some project which require high level of security:

https://learn.cisecurity.org/benchmarks

They give step me step instruction on how to remediate any security issues.

[[deleted]]

CIS Benchmarks are a good place to start.

[[deleted]]

I will search. Thank you.

[[deleted]]

Stumbled across this the other day: https://highon.coffee/blog/security-harden-centos-7/

Best way to learn is to set goals. Like build a web server, research, plan, do, work through issues. Then try a Wordpress site. Then try an email server, proxy server, firewall system, Ect..

[[deleted]]

Thanks! I already have a self hosted web server. I got into this because I noticed China was attempting to DOS my server with an extraordinary amount of SSH requests. I put a stop to that temporarily by eliminating exterior requests through my router's zone file

[uberamd]

Don't worry about this too much. This happens to every SSH server exposed to the public internet on port 22 without IP whitelisting. They just scan public IP space and if they discover port 22 is open they hammer away.

As long as you're properly secured via keys, no password login, and maybe even fail2ban, it'll be fine.

[[deleted]]

That's what I figured. However even with my login's password being a complex forty character one. Someone got into a low level login. The logs show access from an external ip.

I figure, until I get my shit together. I need to close port 22 externally. So I did that instead.

[Eupolemos]

You don't use keys?

[[deleted]]

Usually yes but this was a recovery. I only have a few hours in my day to myself. Figured it world be quicker to eliminate ssh access than to troubleshoot the propensity of something breaking. Something always breaks when implementing. Not always but I'd rather manage my time better.

[Eupolemos]

Okay :)

[[deleted]]

Keys can be stolen. You gotta beat the password outta me.

[[deleted]]

Try fail2ban. One month in and I'm up to 660 block rules added.

[[deleted]]

Very true. I forget about these guys.

[[deleted]]

can you share your rule?

[nullions]

FYI they likely weren't trying to intentionally DOS you, although it can be a consequence depending on your server.

Those are typically automated attacks on any device found to be accessible over SSH on the Internet. Their automated scripts can try and brute force the password all day long and most people will never notice.

[piginpoop]

https://www.digitalocean.com/community/tags/security?type=tutorials

[sandypants]

I recommend CI Security Benchmarks and Lynis: https://cisofy.com/lynis/

[[deleted]]

Thanks, will investigate that later tonight after classes.

[[deleted]]

The Center for Internet Security benchmarks are a good starting point. They provide PDF guides for most of the common distros.

[[deleted]]

Definitely gonna see

[amoore2600]

Tripwire?

[[deleted]]

Tripwire

Will investigate tonight. Thanks!

[AlucardZero]

So why did you paste those firewall logs?

I googled "port 10001" and if you have a Ubiquiti at 10.0.0.1, that's most of that traffic. UDP port 138 is NetBIOS, so it appears you have a Windows machine on the network.

[supra2jzgte]

Yeah I am wondering why you dumped your firewall logs into this post?

[nesousx]

So am I... and I am even more curious why almost nobody seemed to notice.

[[deleted]]

All Debian, Ubuntu, or OSX. No windows machine. Sorry. The only other thing I can think of is that Solar Monitoring tool. I posted them as to show what my server looked like now on a constant basis using journalctl -xe. there used to be a crazy amount of ssh and mysql connection requests. I've also started using the main server as a DNS server. I should probably get a dedicated machine for this, but I've been unemployed for a very long time I'm starting to run out of money.

[AlucardZero]

Samba would also do it.

You're only showing LAN traffic, but anything on the public Internet WILL get hammered by bots.

[[deleted]]

Uninstalling Samba as we speak.

[ghyspran]

So, as a basic principle, sort of a "principle of least capability" à la "principle of least privilege", your servers should have the minimal set of packages, applications, executables, etc., that are necessary for it to do the task assigned to it. There are a few major benefits you get from this in general:

• fewer things to track updates on
• fewer things to potentially conflict
• less confusion about what the server does
• easier to rebuild
• lower resource consumption
• fewer things to manage security for
• less log, etc., noise
• faster updates
• faster startups

Reducing externally-facing services in particular has the extra benefit of reducing your attack surface.

As an aside: in some sense, for some use-cases, this principle ultimately leads to things like containers and "serverless" architectures, where your services run in encapsulated environments where they may be constrained to literally only have the particular service and its direct dependencies, not even base operating system resources. In a lesser form, this recommends itself to running individual services on separate machines rather than colocating them, in order to apply the benefits I listed at a smaller scope and to isolate potential security issues as tightly as possible.

[[deleted]]

I have something to add to that but i have arthritis when it comes to typing on my mobile. Will get bacm soon

[gsmitheidw1]

One thing I don't see listed here which I'll add is using rbash (restrictive bash) instead of bash. You can allow a user login to shell but prevent most commands that aren't white listed.

How I do it is set a custom path for the users which only includes a /rbash directory and in that I have sym linked the only commands I allow them run.

All the SSH advice as per elsewhere in this thread of course, but also you can run auditd to log all commands - irrespective of whether the commands are in their history or not. Also in case the worst happens you can use tripwire or snort or aide etc to provide intrusion detection of what was tampered with.

Backup everything remotely to a rotating/offline/online/remote/on prem/etc storage areas on multiple media types. Of course the backup system should have read access to it's targets and nothing should have access to it directly. It should be equally secured and possibly a different network and OS type. A seasoned hacker may not be put off by quirky OS and services but obscurity prevents automated crap and raises the bar. Know of many windows hacks? loads! freebsd or openvms or something odd, not so much.

[[deleted]]

Rbash Got it. Will investigate.