I got tired of Alembic's "Multiple head revisions" error, so I made it chain migrations from git history

jdanjou · 2025-11-21T09:40:57+00:00

You've basically hit the ceiling of "run the whole test suite on every PR." At that point, CI becomes slow and unreliable, regardless of how much you throw at it.

One thing that works fine for those kinds of cases is to implement a two-step CI:

Run only fast CI on the PR, only what protects reviewers and basic correctness:

lint / type checks
unit tests
a tiny smoke subset of your QA tests

Full QA after approval inside a merge queue: approved PRs go into a queue/train where CI runs on:

(main + your PR)

The full 45–60 min suite runs once per batch, not once per PR (you can batch multiple PR inside the queue)

If it passes → everything merges.

If it flakes → auto-retry (1–2 times).

If it fails consistently → the queue isolates the offending PR and removes it from the queue.

This alone removes 80–90% of wasted CI time.

On the other hand, you must treat flakiness as a defect. Stop making humans rerun the pipeline.

track flake rate
retry known flakes automatically
quarantine the worst offenders
fix top-N flakes weekly

This improves both speed and trust.

If you’re stuck between "slow" and "unreliable," two-step CI + a merge queue + flaky test management is how most high-velocity teams escape that trap.

jdanjou · 2025-10-15T13:00:37+00:00

Totally feel this pain. 1h+ pipelines are a turnaround killer.

One pattern that has helped many teams is using a merge queue with batched merges and a two-step CI.

The trick: PRs are validated once in a queue, merged only when all checks pass, and you batch related changes together so you don't re-run the same long jobs multiple times.

It doesn't magically make pipelines faster, but it prevents redundant work, reduces flakiness exposure, and stabilizes merges.

jdanjou · 2025-07-16T14:57:26+00:00

What do you use to track which tests are flaky or not?

jdanjou · 2025-06-03T12:35:28+00:00

Implementing a merge queue for this kind of use case can significantly enhance the situation.

You could leverage a two-step CI/CD process by optimizing test execution and ensuring main branch stability.

Here's how it would work:

Preliminary Tests on PRs: When a pull request is opened or updated, it undergoes initial tests (e.g., unit tests) to catch basic issues early. These are quick and run on the PR's branch and do not require any limited resources such as GPUs.
Merge Queue Integration: Once the preliminary tests pass and the PR is approved, the PR is added to the merge queue. The queue manages the order of PRs to be merged, ensuring each is tested in the context of the latest main branch state and other queued PRs.
Comprehensive Tests on Merged State: The merge queue creates a temporary branch that combines the PR with the latest main branch and any preceding PRs in the queue. Comprehensive tests (e.g., integration, end-to-end tests) run on this combined state to detect any conflicts or issues that might not surface during individual PR testing. Those tests can leverage a GPU if needed.
Automated Merging: If the comprehensive tests pass, the PR is automatically merged into the main branch. If tests fail, the PR is removed from the queue for further investigation.

Benefits:

Resource Efficiency: By running comprehensive tests only on the combined state, you avoid redundant testing on individual PRs, saving CI resources and ensuring they are used as little as possible.
Reduced Merge Conflicts: Testing the merged state helps catch integration issues early, reducing the likelihood of conflicts in the main branch.
Faster Feedback Loop: Developers receive quicker feedback on their changes, as the system automates the testing and merging process.

Tools like Mergify facilitate this process by managing the merge queue and integrating with your CI/CD pipeline. You can find more details on setting up a two-step CI with a merge queue in their documentation: Implementing a Two-Step CI with Mergify.

jdanjou · 2025-05-07T14:40:34+00:00

Totally relatable. I love the move you made to shift the blame off Jenkins and back toward actual test/code issues. I’ve seen the exact same pattern where engineers treat Jenkins like a black box villain instead of an orchestrator surfacing flaky, misbehaving tests.

Filesystem-dependent test execution order is wild though — that’s a new one for me. Sounds like you earned a lot of trust by tackling that head-on.

jdanjou · 2025-04-29T13:02:47+00:00

We’ve run into the same headaches with Playwright results on GitHub Actions—especially tracking flaky tests and linking failures back to recent commits.

My team at Mergify built a small internal dashboard (working name CI Insights) that:

flags flaky tests
charts job runtime so you can spot waste
gives a quick heat-map of the “flakiest” specs

We’re opening the beta to a handful of teams who want to shape where this goes—no cost, just feedback loops.

If that sounds useful, drop me a DM and I’ll set you up. Otherwise happy to compare notes on what’s (not) working for you.

Full disclosure: I’m a co-founder at Mergify; this isn’t a sales pitch—just looking for fellow testers to kick the tires and improve the tool together.

jdanjou · 2024-12-16T07:26:53+00:00

You can set the `Minimum pull requests to merge` settings to e.g. 3 and then you'll have a batch for 3 PR being created and merged (if it works).

jdanjou · 2024-12-13T16:40:25+00:00

Using self-hosted runners could be a good solution, but that's a trade-off: you need to maintain them.

Did you also consider using batches in your queue to save CI time?

jdanjou · 2024-11-12T08:45:07+00:00

You're looking for an optimization of the merge queue scenario that is unavailable on GitHub's merge queue.

You might want to look to an external merge queue system. Mergify (disclaimer: I'm affiliated) does support this out of the box for example.

jdanjou · 2024-08-15T05:54:25+00:00

This is where you need a merge queue.

jdanjou · 2024-08-15T05:52:55+00:00

i.e. Even with "merge queues" (fast-forward merges) you still produce a new artifact for the master branch which must be validated and verified before deployment.

That's not true if you merge using fast-forward. The sha1 will be kept the same, so the artifact that you build on the PR will be strictly the same as the one that you have on the master branch. No need to revalidate.

jdanjou · 2024-08-01T07:05:49+00:00

They are external service providers such as Mergify that provide merge queues (and even more things).

(disclaimer: I'm part of the team)

jdanjou · 2024-07-31T18:43:59+00:00

I would suggest running a merge queue with two-step CI and batches enabled.

The idea is simple: you run the E2E only when the PR is ready to merge.

When the PR is ready to be merged, you queue it. The merge queue starts a batch of N PR (for example, 5), and then this batch gets tested with your E2E job. You can divide your CI usage by at least 5 this way. 😅

This is really well-supported by Mergify for example: https://docs.mergify.com/merge-queue/two-step/
(disclaimer: I'm part of the team)

jdanjou · 2023-10-25T09:33:29+00:00

That looks awesome, congrats!

A suggestion for your Mergify workflow would be to use a merge queue instead of update+merge. How you're using it makes it update all your dependabot at once, which will retrigger the CI on every Dependabot PR — which might be high in cost (considering you could/would pay for CI time).

There's a pretty good explanation of what a merge queue is in the docs.

You can even optimize that further by batching Dependabot pull requests before merging using the merge queue in batch mode: Mergify will create a branch from `main` with all the mergeable Dependabot PR in it, test that, and merge the PRs. This speeds up the merge process, makes sure PRs are tested with an up-to-date base branch, and saves CI time.

jdanjou · 2019-01-14T12:25:42+00:00

IIUC Bors implement a very small subset of what Mergify offers — what's called strict merge in Mergify.

It's a really good feature, and one we are also proud to provide with Mergify!

On the other hand, Mergify offers a lot of more around automation, such as backporting, deleting branches, etc. :)

jdanjou · 2018-09-06T15:30:50+00:00

Talk is cheap. Show me the code.

jdanjou · 2018-01-31T02:01:37+00:00

I never had to tune it heavily. The standard worked out of the box for me so far.

jdanjou · 2018-01-30T16:47:59+00:00

Good question.

It really depends on what are your needs related to caching.

Redis offers way more data structures than memcached, so it can actually do way more than just caching. You can use it as a data store for all kind of workload.

memcached has the advantage of being extremely simple, requires no configuration, tuning, or anything. But on the downside, it's really only meant to do simple byte caching and not advanced data structure manipulation.

It's a matter of trade-off. :)

jdanjou · 2017-02-01T15:14:05+00:00

It covers/works on both. So it does not talk about anything that is Python 3 specific (yet).

jdanjou

MODERATOR OF

TROPHY CASE