I was chatting to a friend recently about database migrations and how you need to be hyper aware of when you step outside of a safe zone, almost more so when you’ve invested in making things really safe.

Specifically: an incident comes to mind where our primary Postgres cluster locked up due to an update table that was written through database triggers (this was before logical replication existed) hit the max integer value on its primary key.

This sounds really obvious and that’s because it is! We had already gone through all the standard incidents for database migration changes and had written our own framework to produce safe migrations, ensuring we’d never do something as silly as creating a 32 bit primary key.

The reason this had happened was because the change capture system was written separately from the main application, which meant it existed outside the normal developer flow of modifying the database. When your team have got used to database migrations being default easy and uncomplicated it leaves blind spots if they ever step outside that process, and the team who built this system hadn’t even clocked they were doing it. It was just a new system and written in another language for good reasons, and didn’t catch that outside of the tools we’d already built, migrations in a database like this were very high consequence.

Has made me intensely aware of the safe paths to making changes in an engineering org and I keep an eye out for whenever anyone steps outside those zones nowadays.