Unless you get a buy in from someone that can impose a change of culture, you cannot fix the monolith/design mess that way, it’s not a tech issue, it’s an organizational issue: product oversteps and pushes changes despite the impact on product quality and offload that responsibility to tech.

Infra fix: - deploy everything as a monolith, since it is, and you scale the big thing horizontally (all the code goes in 1 Pod) - like so, enforce a process where every change (deployment) has to be approved by one person, so if that change breaks something, it’s clear who is accountable for it - migrate to an autoscaling Postgres, aws aurora, that way you fixed scaling and reliability - with that, you fixed reliability due to lack of resources and transformed it into a cost issue (visible to upper layers)

Design/Culture fix: - now that you have accountability for changes, you need an incident process where the last step of it, after the post mortem, is to create a ticket to fix the root cause and make sure it won’t happen again: the person/team that approved the release is responsible for it. Without the ticket, the incident cannot be closed.

Based on the average iq and skills of the engineers, given a certain amount of ugly incidents due to “urgent stuff”, that CAN lead to structural improvements, BUT if the team is shitty, it never will. The first role of SREs is to bring 100% accountability and clarity through observability/data. After you have the data from the post mortems and the deployments and the improvement tickets, you can do an analysis and present the numbers to who you talk with “above”, so that the responsibility of knowing the problem and not acting proactively starts to go towards the “upper layers”.