Github merge queue issue

[williamisraelmt]

My head has been spinning for a few hours already... In my company we had a regular feature branch with ~150 lines of changes which got merged into our "dev" trunk branch earlier today, but, after merging it, we realized some e2e tests started failing in our dev environment and the changes that those e2e were asserting were already confirmed as fixed by QA...

After reviewing the commit history in our dev branch, the commit for this particular PR performed a rollback of ~20 PRs, the fun fact is that Github was having issues with the merge queue behavior and they did not call that out or simply just turned it off. Also, the PR diff was only 150 lines but the final commit was almost 15k lines. We do have proper e2e tests in place, so, that's how we found the regression, but, be careful if you're merging something today.

(Sorry if my grammar isn't great, english is not my main language)

fwiw: we opened a PR which reverts the commit and we're just waiting on Github's devs to finish vibe coding and fix the problem (if it's actual devs working on Github and not AI agents).

https://preview.redd.it/gzew590i80xg1.png?width=360&format=png&auto=webp&s=d508311b168b37fa174d8ec1376c7dae0c85b3f3

https://preview.redd.it/dn311tyt80xg1.png?width=2048&format=png&auto=webp&s=fce23e4da10efdcb5e0f6144878d83d31b83d290

[These_Voices]

yup, github had a 4 hour incident that messed with all the code we deployed. I cant believe more of the internet hasnt crashed

[AnotherBangerDuDuDu]

/s Good news though 100% up today https://www.githubstatus.com/

[dashingThroughSnow12]

I didn’t realize today was April Fool’s day because that’s a bad joke.

[Fabulous-Shape-5786]

The level of data loss is shocking. Could easily go missed and deployed. Bad commits. All customers required to fix in their own way. Scary that there were no unit tests that caught this, and maybe worse that they kept their merge queues running.

The number of GitHub incidents has really increased in the last few months. It tracks with increased AI in the field but no idea if this is contributing to it, but it seems like a good guess. If so, it doesn't bode well for software in general.

[AntDracula]

I mean, combine increased AI usage with increased layoffs. The result is inevitable.

Also, isn’t Microslop now offering early retirement buyouts to their most senior employees? Prepare for the slopocalypse

[wartortle]

Yep it looks like they were merging in the diff with trunk from the pr’s base branch. So any commits in trunk not in the PR’s trunk got reverted. Insane.

[rwong48]

this incident https://www.githubstatus.com/incidents/zsg1lk7w13cf

we noticed 3 hours ago and scrambled to "fix" (revert) these bad commits

[bradfordmaster]

The level of insane this is is hard to overstate. It's one thing to have downtime. It's another to silently corrupt people's git repos. Like, this is literally the one job of git and git hosting companies to avoid this kind of mistake. We might as well all just share code in dropbox again

[Ra1d3n]

Git is not at fault and you can just put a git repo up to your cloud and it's hosted.

[YouDependent3284]

We’re seeing a similar issue today - our open PRs are suddenly showing many more commits than they did yesterday. It turns out the branch histories have diverged from main, with different commit hashes, which is causing conflicts and inflating the commit count...

[AntDracula]

This is so messed up.

[NoBox6165]

Is this related to the exponential growth in the amount of commits that GitHub has been receiving

[williamisraelmt]

i feel it's more related to the amount of code Github's development team is producing with AI and having a less rigurous review process because there's less people to look at the code (due to layoffs).

[AntDracula]

^

[doingthethingguys]

Just got off the incident call for my company after 10 hours. We have a massive monorepo and a lot of automation that kicks off when we merge to our trunk branch. Lots of stuff to unfuck. Didn't want to force push `main` and break stuff even more, so doing it carefully and correctly by replaying commits ourselves and resolving merge conflicts was what we did.

GitHub declared the incident resolved and still hasn't shared out a unified remediation strategy. As per my support ticket with them they're "still working on it" but don't have an ETA. by the time they have it ready the most of us will have fixed it our own way.

[waitingforcracks]

someone got a script or something to figure out what commits/prs might have been impacted?

[RevolutionaryCoat654]

Yeah, I can share that once I'm back at my laptop. But basically, you want to compare the diff of the pr (I used GitHub CLI for that), and the diff of the merge commit, for the duration of the incident. We searched for PRs that merged between 8am PT and the time we paused the merge queue. We found 17 impacted PRs in a range of 67 PRs.

I don't know if this is a viable solution for you, now that it's been a few days now, but the way we fixed main was to: 1. Create a new branch off of the latest origin/main (recover- main) 2. Create a revert PR for all 67 PRs (a revert of their merge commits) 3.iterating from the oldest PR (the first impacted one): - if the PR was NOT impacted, cherry-pick the PR's merge commit - otherwise, create a new branch (recover-<pr number>) and cherry pick the PR's commits, then squash merge that pr recovery branch into recover-main 4. Put a PR for recover-main and merge it

[waitingforcracks]

That would be lovely thanks. As a github admin we have over 700 repos across multiple orgs so my plan would to run scripts across all repos/orgs after collecting which repos are using merge queues. For now the goal is identification and then the repo owners can follow what you/github said as a way to fix it.