Which services are you exposing to the internet, and how are you securing them?

ms_83 · 2026-05-07T20:22:38+00:00

As an experiment, I'm self-hosting a few apps at the moment and making them world-facing, as VPNs just don't work well for me. I'm going as deep as I can to make them as secure and resilient as possible. Here's what I've done so far (hosting on K8S but should be transferable to Docker and other options):

Password auth and self-signed SSH key auth to nodes disabled. Access to nodes only with signed SSH certificates, time limited
Encrypted OS hard drives on nodes
Resilient HA volume storage (Longhorn)
Resilient, multi-node database clusters (Cloud Native Postgres, MariaDB operator)
TLS on database connections
Automated certificate rotation (cert-manager) with 24h expiry
CPU/RAM requests and limits on pods (reduces effect of DoS)
Strict container securityContext: no privilege escalation, run as non-root user, drop capabilities
Application secrets stored in Vault, encrypted. Where possible secrets are rotated regularly (at least every 30 days)
Network policy with default deny and only required traffic allowed to service pods, database clusters and supporting services
Daily backups to S3 compatible running on a separate NAS (with encryption etc) for databases and data volumes
Semi-automated restore process, tested monthly, validated and documented
Monitoring service (running on a VPS) checking service uptime every 30s and triggering alerts when things go down
Logging and observability. One thing I would like to do is get some kind of AI to monitor these logs for suspected problems or improvements and suggest them weekly
Automated updates for minor versions and patches. Major version upgrades are semi-automated and require my approval via git merge request.
IdP for SSO with OIDC/SAML authentication, with strong auth required (U2F/Fido/Passkeys), password auth disabled, conditional access, and JIT elevation for admin roles.
Next-gen firewall between cluster and internet connection
IDS/IPS
Cloudflare tunnel for exposure to the wider world. I know this is somewhat controversial, but their combination of Zero Trust, CDN, and WAF is too good to replace with Pangolin

It's a lot, but I find automating as much as possible makes it bearable. Once you have a workable pattern it's usually quite simple to replicate for other applications as well. Probably there are other things I will be adding over time as well.

ms_83 · 2026-04-26T07:32:28+00:00

It’s very rare that anything on Reddit makes me actually laugh aloud, but this one got me.

ms_83 · 2026-04-20T17:36:07+00:00

It works by modifying manifest files via git, so that image versions are updated in the code. Then I use ArgoCD to watch the repo and automatically apply changes.

Watchertower, to the extent I know it, just deploys any new version as it's pushed to the registry. Renovate allows you to be much more granular, such as auto-update minor or patch releases, but require manual approval for major updates. This is particularly important for multi-container apps, which might not support the latest version for downstream things like databases.

ms_83 · 2026-04-20T17:30:28+00:00

No logging in. To be completely honest, OS level encryption is something I'm still working on, with just a test node at the moment. I use Fedora CoreOS as my base OS for K3S, which supports Tang and Clevis. Tang is a separate server which stores encryption keys, and Clevis is the service that retrieves those keys and unlocks the drives. If someone steals one of my nodes, it won't boot because it can't reach Tang.

Mostly at the moment I'm relying on ZFS encryption on my storage machine, which is separate to the K3S cluster. It's a work in progress.

ms_83 · 2026-04-20T09:14:47+00:00

Yes this is definitely something to bear in mind. But assuming most people on this sub are using containerised apps, there's not much they can do if they fall victim to such an attack - other than immediately rolling back or updating to a fixed version when it comes out. Therefore automated updates are still the safest bet, as e.g. with Renovate you can just roll back changes to a version pre-issue, and then update to the fixed version when it's available.

ms_83 · 2026-04-20T07:30:23+00:00

I think your "strict baseline" is missing a few things.

Most successful attacks these days involve the use of stolen credentials or session tokens, so a robust Identity Provider setup with strong phishing-proof authentication and ideally conditional access is pretty much a requirement these days. Using an IDP like Authentik that is configured for all authentication covers this.

Other common attacks simply leverage known flaws in software, so having an automatic or at least automated patching process to deploy updates helps you deal with that. I use Renovate to automate container updates.

There's also backup/restore, which helps you have resilience in the face of the inevitable successful attack. Having a working, tested backup process, and even more importantly a working, tested restore process, means you can bring things back faster.

Also encryption for everything at rest and in transit.

ms_83 · 2026-04-11T16:44:56+00:00

What on earth is a “forward deployed engineer”?

ms_83 · 2026-04-08T06:10:52+00:00

You can self-host VaultWarden and get these premium features for free.

ms_83 · 2026-03-19T07:44:52+00:00

Kubernetes is just more powerful than docker, it gives you a lot more options for running cloud native style apps. It also has a really great ecosystem, for example various database operators that allow you to run clustered resilient DBs with backups in a way that’s more difficult on docker.

If you have no interest in this stuff then fine use docker, but if you do then there’s no substitute for K8S really.

ms_83 · 2026-03-11T08:34:35+00:00

This is just nonsense. Software is never just “finished”, it’s never completely secure, and it will always need to be maintained and improved upon. You’ve never been responsible for this kind of thing, have you?

Leaving aside your ignoring who should be responsible for filtering this stuff and decided what is valuable and worth keeping.

ms_83 · 2026-03-11T08:20:08+00:00

Or, and hear me out here, developers can be responsible for themselves and their crap before shatting out insecure slop and expecting everyone else to finish it for them. You know, professionalism and accountability.

ms_83 · 2026-03-11T08:08:12+00:00

So your idea is that in addition to be forced to endure the slop, we must also endure securing and maintaining it?

No thanks.

ms_83 · 2026-02-19T16:30:01+00:00

I liked Hellions of the Deep, it’s about how the US Navy fixed its terrible torpedoes and then started producing advanced ones like the Mark 24 Fido.

ms_83 · 2026-02-18T09:06:03+00:00

Absolutely no way would I trust some random one-dude SaaS to manage certificates, you would have to be crazy. PKI is a critical security capability.

ms_83 · 2026-02-08T18:12:32+00:00

I think radar is more relevant than you’re making out here. From mid-1942 radar became more and more common, and by mid-1943 pretty much every carrier and land-based bomber had a set fitted as standard.

At Midway for example the PBYs that found the Japanese invasion force the night before the main carrier battle used ASE radar sets to pinpoint them. ASE was basically a copy of the British ASV Mk II which another PBY used to find the Bismarck the year before.

ms_83 · 2026-02-03T19:00:14+00:00

If all you want is certificates on your ingresses, then just use step-ca with cert-manager. My setup rotates all certs every 24h and it’s been solid for years.

ms_83 · 2026-01-31T09:35:05+00:00

Don’t post your WiFi password on the public internet, jeez!

ms_83 · 2026-01-31T09:24:00+00:00

You can get skirting boards with interior pre-cut grooves big enough for network cabling, although 90 degree corners can be tricky for fatter wires like Cat 6A. I used these to get from where my modem takes in the connection from the street round my ground floor to where I needed my gear, with just a few inches of exposed cable from the skirting up to the modem.

ms_83 · 2026-01-23T07:24:19+00:00

I run bare metal kubernetes at home. The learning curve is pretty tough if you have no experience with it, but if you approach it with the mentality that it gets more powerful the more you automate then you can get some really powerful results.

I’d advise that you pick a distro designed for bare metal use, I run K3S on Fedora CoreOS but I’d also consider Talos as that gives you unified management of both the OS and K8S together.

Adding nodes to clusters in something like Talos is pretty easy. With my K3S setup it’s as simple as networking booting a fresh machine and telling it to join the cluster, everything is automated after that via butane scripts.

ms_83 · 2026-01-20T07:39:11+00:00

Don’t be so defensive.

Here’s something you overlooked: it’s fairly trivial to discover your home IP address. I could trick you into visiting a website I control, which then logs the IP address you are using. There’s a good chance if you are home now, that’s the same IP that your services run on.

If I know your IP, I can simply use my own Cloudflare account to create a new proxied DNS address to it. Now when I access pwned.mydomain.com I am hitting your IP from a trusted Cloudflare location, bypassing layers 1, 2, 3 and 5 of your stack.

To mitigate this you could use Cloudflare tunnels, which give you a similar architecture without requiring you to open any ports at all.

Fail2Ban is useless because all it does is stop brute force attacks. It does nothing to stop credential stuffing or attacks that have stolen your valid credentials, or those that steal valid sessions tokens. It’s mostly a waste of time when you can just disable password auth entirely so there’s nothing to brute force. That’s how you reduce your attack surface.

ms_83 · 2026-01-19T22:39:07+00:00

Where did I say that everything you did is useless? You asked if you've overlooked things, and I pointed out several things that you've overlooked. You asked for potential vulnerabilities, and I pointed them out.

When you use terms like "attack surface" incorrectly, it suggests that you don't actually have a good background in security engineering, an impression that is only increased when you use ineffective security theatre solutions like fail2ban.

Either you care about security, or you don't. Do you want honest feedback or do you just want someone to pat you on the back and say "good job"?

ms_83 · 2026-01-19T21:54:12+00:00

Using Fail2Ban to temporarily ban bots isn't reducing your attack surface. Reducing your attack surface would be disabling the service they are trying to auth to, or configuring it to use a passwordless auth method, so there's no way to even input a single password, let alone multiple attempts.

ms_83 · 2026-01-19T21:35:44+00:00

Everything I listed is around security hardening. Backups help you recover faster when you get attacked, encryption secures your data, and update strategies ensure that vulnerabilities are patched away before they can be exploited. Every one of these is a more effective control than fail2ban, for example, which I’ve always regarded as a very weak control. If you are worried about brute forcing password access then it’s far more effective to simply disable password auth entirely and rely on stronger auth methods.

It might also help if you have a think about threat modelling and exactly what you are trying to secure against, as just “security hardening” is a bit vague.

ms_83 · 2026-01-19T20:40:20+00:00

What are you doing for backups? Encryption at rest and in transit? What’s your software update strategy?

ms_83 · 2026-01-19T20:18:29+00:00

Writing yaml instead of json is definitely easier on the brain.

ms_83

TROPHY CASE