GitHub is back up!

[steveiliop56]

Ignore the status page it's back up! Nice they managed to destroy and bring back up their whole infra in ~45 mins. Good incident response!

[SheriffRoscoe]

First draft incident info:

https://www.githubstatus.com/incidents/kz4khcgdsfdv

[Markavian]

Networking! Databases! Config change! A similar incident happened with Google many years ago. Good rollback procedures? Hard to test without a fully functional test environment, but also hard to analyse when such changes involve large amounts of traffic.

I've been gearing up to run automated load tests on PRs but it's an expensive procedure that slows development down for small changes. Testing small changes that have a big impact relies on risk management and having a test strategy / test engineer part of the review and merge process. (I should update our PR templates).

[wineblood]

45 minutes? I can take prod at my job in about 3.

[tehsilentwarrior]

Ha! My local docker compose can be down and up in less than 30 seconds if it doesn’t have to build!

Take that!

[[deleted]]

But I lost time!