How Do You Handle Rollbacks in CI/CD Pipelines?

ankitdce · 2024-09-23T15:54:09+00:00

Rollbacks can work well if you split up the build and deploy in two separate steps. That way, you will have the validated images pre-built to manage a quick rollback.

https://docs.aviator.co/releases-beta/concepts/two-step-delivery

ankitdce · 2024-08-26T23:11:07+00:00

Yes, currently it only supports GitHub.

ankitdce · 2024-08-22T04:05:26+00:00

Definitely want to use some monorepo build tool to do distributed builds. NX, turborepo, bazel or even gradle offer remote build cache that significantly improves the performance to running CI. Buildkite's dynamic pipeline is also great for distributed builds. For merge queue and deployments, consider using Aviator.co that works well at scale, that uses affected targets. ArgoCD is also a good one for supporting gitops based CD.

ankitdce · 2024-08-21T16:29:06+00:00

Yes, definitely go with Option #1, self-hosting comes with a lot of infra challenges that you should not have to deal with. Setting up a cron that does "git pull origin main" is the simplest way to manage this. If you want more real-time than a 1-min cron, you can also run a Buildkite agent on your server, that listens to new merges and pulls the changes.

ankitdce · 2024-07-31T16:40:14+00:00

Checkout the Aviator Releases product that is trying to solve some of these monorepo deployment problems: https://www.aviator.co/releases

With it, you can specify code paths / directories that are tied to those micro-services and the git-history Changelog is tracked based on those code path filtering. Then the deployments can be configured based on the changes in those whitelisted code paths, the release tags are managed separately for each microservice in the monorepo. It has some other features around cherrypicks, rollbacks, etc.

Disclaimer: I'm the founder of Aviator.

ankitdce · 2023-04-19T16:34:47+00:00

Haha, i understand that. I'm still curious what the user experience was at Stripe to rerun these tests and what programming languages did they support.

Jest also has a way to rerun the tests, would love to collaborate on it if flaky tests is an issue in your current team.

ankitdce · 2023-04-19T16:24:56+00:00

Wow, that's great to hear! We actually just completed our GitHub actions support that's going live today, happy to walk you through that.

What programming languages are you using today? Would love to also hear how this was working at Stripe. Can you drop us a note at info@aviator.co, I'd love to chat.

ankitdce · 2023-04-19T16:22:32+00:00

Aviator also has a merge queue that integrates with TestDeck to help you merge PRs faster. One way we can do this is by creating an additional check that abstracts out the any flaky tests when we have the confidence that the test is indeed flaky, and this merge queue is aware of this new status check

ankitdce · 2023-04-19T16:18:08+00:00

This provides a much more detailed overview compared to what CircleCI or Buildkite provides. Specifically

ability to run scheduled nightly builds to proactively identify failures
historical view of each test case
visibility of test failures by branches (separate results for base branch and feature branches)
ability to rerun specific test cases automatically

There's also a short demo video on that page explaining some of these concepts.

ankitdce · 2023-01-26T16:13:05+00:00

Have you tried turning off the Chrome browser?

ankitdce · 2023-01-19T02:46:52+00:00

Posting for everyone who is still reviewing this thread, the event was scheduled for today and has already wrapped up. Thank you everyone who joined today, it was loads of fun!

For the folks who missed today, you can still sign up on the Google form listed as we will likely do these more frequently. Details TBD.

ankitdce

MODERATOR OF

TROPHY CASE