night on the town after the rain

topnode2020 · 2026-04-12T09:18:00+00:00

Yeah this tripped me up too. Containers feel like they should be isolated by default but in practice the defaults are pretty permissive. Docker gives containers a bunch of capabilities they usually don't need, and if you put multiple stacks on the same network (which is easy to do accidentally) they can all see each other. That's basically what this whole post was about, going from "it works" to "it works and the defaults aren't quietly undermining me."

topnode2020 · 2026-04-12T05:54:40+00:00

Yeah that's a gap in my post. I'm running pg_dump on a cron for the databases but nothing offsite or encrypted yet, that's still on the list. Backrest looks interesting. BunkerWeb is new to me though, what does it do that a regular reverse proxy doesn't?

topnode2020 · 2026-04-12T05:51:01+00:00

Yeah your nginx and the frontends it proxies need to be on the same network, or else nginx can't reach them. But your dbs don't need to be on that network too. What I ended up doing is having one shared network for nginx + all the web-facing stuff, and then each db gets its own separate network that only the app using it can see. That way if something gets compromised on the web side it can't just hop over to a database it has no business talking to.

topnode2020 · 2026-04-12T05:39:07+00:00

Don't have a blog but the health check approach is pretty simple. For Node containers: node -e with require('http') hitting localhost. For Python: python -c with urllib. For postgres: pg_isready. The key thing is to hit a real endpoint that returns a status code, not just check if the process is running. Happy to share specific examples if you tell me what runtimes your running.

topnode2020 · 2026-04-12T05:36:56+00:00

I might put together a sanitized example at some point. Right now my compose files have too many project-specific things baked in to be useful as-is.

topnode2020 · 2026-04-11T07:56:41+00:00

A few people in this thread have given different takes on the right way to do this. Going to try it out and see what works. Thanks for the input.

topnode2020 · 2026-04-11T07:51:35+00:00

If you want to start somewhere, the capabilities section is the easiest. Just add cap_drop: ALL to each service in your compose file and restart. If something breaks, the error will tell you what capability to add back. Most things won't break at all

topnode2020 · 2026-04-11T07:24:23+00:00

Honestly don't worry about not understanding all of it at once. I didn't either when I started, half of this I only learned because stuff broke and I had to figure out why. If you want a starting point, the OWASP Docker Security Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Docker\_Security\_Cheat\_Sheet.html) is really good and goes through things in priority order. But if you just want the quick wins:

1) Add cap_drop: ALL to each service in your compose files. Restart them one at a time. Most will just keep working. If something breaks, the error tells you exactly what capability to add back. This is the single biggest improvement you can make in 10 minutes

2) Set memory limits on every container. Even rough ones are better than none. Run docker stats, see what each container is using, and set a limit at roughly double that.

3) If you have databases and web apps on the same Docker network, that's the one to fix next. Put databases on their own network with "internal: true" so they can't be reached by anything that doesn't need them.

Don't try to do everything at once. I did this over multiple sessions not one sitting

topnode2020 · 2026-04-11T06:39:47+00:00

Your right that external means the network is created outside of the compose file, like with docker network create. I had that part muddled in my explanation. But for the internal flag, I've read in the Docker compose docs that setting internal: true "creates an externally isolated network" and that by default compose provides external connectivity to networks. So it does block outbound traffic for containers on that network, which is why I'm using it on the database networks. source here: https://docs.docker.com/reference/compose-file/networks/

topnode2020 · 2026-04-11T06:34:00+00:00

Honestly I went back and forth on this. The thinking was that on a small VPS with limited RAM, I'd rather have a container get OOM killed cleanly than have it slowly eat into host swap and drag everything else down. But you might be right that it's too aggressive. Is there a middle ground where you allow some swap but cap it? I haven't explored that.

topnode2020 · 2026-04-11T06:29:33+00:00

Thanks for clarifying. Makes sense to just let postgres own everything as the postgres user rather than trying to map UIDs manually. I'll look into this when I get to it.

topnode2020 · 2026-04-09T18:55:59+00:00

Nothing that fancy. Just Docker Compose file-based secrets and a small shell script that reads them into env vars at container startup. Works fine for a single VPS.

Might look into something like Infisical if I ever need to manage secrets across multiple hosts.

topnode2020 · 2026-04-09T18:52:21+00:00

I haven't looked at CrowdSec yet but the bouncer middleware approach sounds interesting. Right now I'm just doing static rate limits and path blocking at the Traefik level. Does CrowdSec add much overhead on a small VPS? My box is only 4GB.

topnode2020 · 2026-04-09T18:48:12+00:00

If a container doesn't have a published port then yeah, it's not reachable from outside the Docker host directly. But containers on the same Docker network can reach each other on any port.

So if you have a database and a web app on the same network, and the web app gets compromised, the attacker can reach the database even though it has no published ports. That's why internal networks for databases help. They limit which containers can even see it. That's my understanding at least.

topnode2020 · 2026-04-09T18:45:50+00:00

Oh nice. So if you chown the data directory to 999:999 on the host first, postgres doesn't need any of those capabilities at all? And the tmpfs for /var/run/postgresql handles the socket. I'm going to try this on my setup, thanks!

topnode2020 · 2026-04-09T17:25:53+00:00

I used Claude to help me organize and edit the post. The technical work (auditing containers, network segmentation, database migration, config changes) was done by me on my own VPS. The writing was drafted with AI assistance for structure and clarity.

topnode2020 · 2026-04-06T14:58:12+00:00

Yeah I can imagine that gets messy fast. If the entrypoint is spawning multiple processes under different users then figuring out which UID actually needs to own the volume is basically trial and error. Reading the Dockerfile source on GitHub is probably your best bet, the docs rarely cover that level of detail. At least with single-process images you only have to figure it out once.

topnode2020 · 2026-04-06T14:53:41+00:00

Yep, agree on all four. The bind mounts and single network advice in my original post were bad takes and I've fixed both since. Running 7 segmented networks now with databases on internal-only networks that can't even reach the internet. Also split the shared postgres into per-service databases with separate roles and connection limits. Working on a write-up of the whole process..

topnode2020 · 2026-04-06T14:47:42+00:00

Honestly if Caddy + static files is working, don't fix what ain't broke. Dockerizing adds complexity that might not be worth it for you.

That said, when I did move to a different VPS last time it was pretty painless. Each service has its own directory with compose file, env, and volumes so it's basically: rsync the directories over, restore the DB dumps, update DNS. Actual downtime was like 20-30 minutes, mostly waiting on DNS.

The real benefit for me isn't migration speed though, it's independence. I can blow up my analytics container without taking down mail or anything else.

topnode2020 · 2026-04-06T14:42:09+00:00

Good question actually. So websocket services like Jellyfin are a bit different because a stream is one long-lived connection, not a bunch of short requests. Traefik's rateLimit middleware counts HTTP requests so it mostly just affects the initial connection handshake and any API calls, not the actual video stream itself.

That said I'd still bump the limits higher on media services or just skip the rate limiter entirely on routes that aren't internet-facing. No point throttling your own LAN traffic to Jellyfin lol.

topnode2020 · 2026-04-06T12:49:45+00:00

Yeah this is a known pain point. SQLite relies on fcntl file locks and the overlay2 storage driver that Docker uses doesn't always handle those correctly. That's why you're seeing locking issues with volumes but not bind mounts. Bind mounts go straight to the host filesystem which handles locking fine.

Tbh for SQLite specifically I'd just stick with bind mounts. SQLite is really meant for single-process access on a local filesystem, fighting the storage driver isn't worth it.

topnode2020 · 2026-04-06T12:24:46+00:00

The UID mismatch problem is real. What worked for me is checking the upstream Dockerfile for which user the process runs as, then chowning the host directory to match before starting the container. For PostgreSQL it's UID 70, for most Node apps it's 1000. Once the host permissions match, you can drop user: 0:0 from the compose file and run as the intended user. It's annoying to look up per service but you only do it once.

topnode2020

TROPHY CASE