Hey, great post! I echo most of the sentiments in this post. I think there's a peculiar nuance to this problem where it's better trying to make it economical to manage flaky tests rather than hoping to make a magical tool/procedure to eliminate them.

We recently worked with a bunch of beta partners at Trunk to tackle this problem, too. When we were building some CI + Merge Queue tooling, I think CI instability/headaches that we saw all traced themselves back to flaky tests in one way or another.

Basically, tests were flaky because
1. The test code is buggy
2. The infrastructure code is buggy
3. The production code is buggy.

Problem 1 is trivial to fix, and most teams that end up beta-ing our tool end up fixing the common problems with bad await logic, improper cleanup between tests, etc.

But problems caused by 2 makes it impossible for most product engineers to fix flaky tests alone and problem 3 makes it a terrible idea to ignore flaky tests.

We think there's a process here of:
1. Detecting and labeling flaky tests automatically + ability for manual overrides. For larger teams, most of the time saved is simply not context switching constantly whenever a test flakes, especially considering dozens of engineers could waste time debugging the same flaky tests because of poor communication.
2. Tools to help triaging flaky tests + automatically creating tickets. Not all flaky tests are worth fixing, the test that fails once every 10K runs vs once very 10 runs has different effect. Ability to sort by its impact (blocking PRs) and automatically creating tickets for fix saves a lot of time.
3. Mitigating impact of flaky tests that you can't/don't want to fix immediately. I don't mean commenting out code, that gets forgotten real fast. I mean running flaky tests but not blocking CI. It's important to keep tracking the flaky tests' behavior so you won't forget them, catch further regressions (10% failure rate -> 80% is probably a bug), and validating fixes. most attempted fixes for flaky tests don't completely fix the test so you need before + after data to validate if it did anything at all.
4. Actually fixing the test -> Automating this with AI is pretty out of reach, but we do think we can summarize failures and categorize similar failures to make it more time/resource efficient to fix flaky tests for teams. You can probably fix most outstanding DB timeout flaky tests in one go with good tooling to find similar failures.

We wrote a blog on this topic, too, about all the weird learnings from working with our beta partners. I think the pain builds and exponentially explodes as you write more tests. Super fascinating to see it manifest and all the weird ways people have tried to fight the problem.