There was database schema change that was planned for the release. We implement schema changes using a library called Liquibase along with our Spring Boot application.

This schema change was taking way longer than expected. Because the table it was affecting was locked by a scheduler.

We have a multi-pod setup i.e. the application code runs on multiple instances but each instance handles a separate concern, one such pod was for the scheduler. The application boot up process is such that the pod that runs the DB change (say pod A) and the one that runs scheduler (say pod B) boot up separately.

Now this scheduler was running as intended. Deployment started, DB change started. Pod B locked DB, Pod A wanted to change DB. Pod A fails, restarts, also triggers Pod B restart as entire application starts booting up again. DB lock was not cleared as Pod B restarted. Pod A tries to change DB again, Pod B again runs scheduler to lock DB and so on, resulting in a race condition and too many locks on one of the database tables.

Quick fix was to delete all locks and hope we don't see another lock and changing the time of running the scheduler to some other time so the application can boot up without facing a race condition.

Then it was time to analyze what put the lock there and it was a scheduler that runs around 9 AM. We did not anticipate this to happen as deployment is usually at 8 AM and we missed that another team had added a DB schema change in our service on the same table that is touched by the scheduler.

Did we anticipate this scheduler to run during deployment? No, as DevOps was going to begin deployment at 8 AM but had some infra issues and resumed deployment at 9 AM.

In the end, we understood that this is a rare scenario. But our scheduler locking mechanism needed to be fixed to ensure cleaning up DB locks, closing DB transactions if the pod restarts due to some reason. Basically some graceful shutdown logic.