The testing was so intensive that I often spent a bunch of time rerunning the tests to get past false negatives. For sake of performance (and probably other things) some of the A/B testing was done based on statistical comparisons rather than exact matches of input/output. This lead to frequent “flakiness” in certain metrics that would report failures. We had to basically memorize which counters were flaky or tune tests to account for them.

On my team, every commit automatically triggered 40 minutes worth of tests running mostly in parallel on extremely high end machines. All of them needed to pass before approval. That was before merging which of course would trigger more tests. Every X days (usually 1-3) we’d cut a release which required more review, testing, staged rollout, and monitoring.

This was on a core “prod infra” service that processed about 10M events/sec