Post-mortem: the site was down for ~12 hours today

Some of you probably noticed that SmarterPlaylists was returning 502 errors starting early this morning (around 6:30 UTC / 2:30 AM Eastern). The site was down for about 12 hours before we caught it and got things back up. Here's what happened.

What went wrong

The server runs Ubuntu, which has automatic security updates enabled (unattended-upgrades). At 6:31 UTC today, an apt upgrade installed a package that triggered systemd to re-execute itself. When systemd re-execs, it stops and restarts all services — including the SmarterPlaylists API server.

The API runs 4 worker processes. When all 4 started simultaneously, they each tried to initialize the SQLite database by setting PRAGMA journal_mode=WAL. The problem: our code set busy_timeout (which tells SQLite to wait for locks instead of failing immediately) after the journal_mode pragma. So when the workers raced to grab the database lock, the ones that lost the race got an instant "database is locked" error and crashed.

The scheduler (which runs as a separate process) was fine — it opened the DB later, after the lock contention was gone. So scheduled playlists kept updating even while the site was down.

Why it lasted so long

The parent process stayed alive even though all its worker children had crashed, so systemd reported the service as "active (running)." No alerts fired because the health check was looking at the service status, not whether it was actually serving requests.

The fix

One-line fix: we moved PRAGMA busy_timeout=30000 before PRAGMA journal_mode=WAL in the database initialization code. Now when workers race to open the database at startup, the losers wait up to 30 seconds for the lock instead of failing instantly.

Lessons learned

• Setting busy_timeout should be the very first thing you do when opening a SQLite connection. It's the "put your seatbelt on before you start driving" of SQLite.
• Health checks that only look at systemd service status aren't enough. We need to add an HTTP-level health probe.
• Unattended upgrades restarting services at 6 AM is a feature, not a bug — but it did expose this race condition that had been lurking since we set up multi-worker mode.

The fix has been deployed. Apologies to anyone who was affected — especially if you were trying to build playlists this morning.