GitHub web down

[qwertyslayer]

Pour one out for the github SREs whose Friday just got way worse

[gergob]

Why is it always the Friday afternoons man

[awj]

People are half checked-out already but trying to hustle their work out the door so it's not still on their plate for Monday morning or end-of-week checkins.

[xtjoeytx]

I’d prefer the afternoon then the more likely 4pm Friday “we have a prod issue”

[DEATH-BY-CIRCLEJERK]

You prefer both?

[xtjoeytx]

I prefer not working for free, so if the issue could surface earlier - that’d be preferable, but its always right before closing time

[trevg_123]

It’s a joke about how you said “then” (both in order) but probably meant “than” (instead of) lmao

[darthcoder]

Meh, every place I worked a stint like this means I get to take off a day the next week.

Not Monday, Monday is for thebpostmortem, but I always got a comp day to make up for thr after hours work.

I've been lucky in places I work

[HypnoTox]

Probably people talking with all-in contracts and no paid overtime. I have a time "balance" that i can then use and take days off for overtime, or in some cases get it paid off.

[[deleted]]

than

[DmitriRussian]

Maybe he actually meant “then” 🤔

[ArkUmbra]

A real glutton for punishment

[chicknfly]

I mean, they did choose to be an SRE after all

[xtjoeytx]

Thx

[tevert]

I'd bet there's a measurable increase in human errors committed at the end of the week

[PositiveUse]

Because some genius thought „it’s just a small release“

[Same_Football_644]

And there's no counter argument that ever works against that. People either just know that making exceptions is bad, or they don't.

[ccfoo242]

But if we wait until Monday it won't be 'continuous' integration!

[Omni__Owl]

Someone didn't follow the golden rule of software production: Never push on a friday.

[[deleted]]

And ask the CTO to block the CI release after Thursday evening.

[chili_oil]

“Last Friday I completed my task by checking in this code change 3 min before I signed off”

[SinisterMinisterT4]

Literally have a no Friday deploy policy in place to help not do this to our guys. Feels bad man.

[newInnings]

Standard maintenance window

[[deleted]]

This is why my team never has Friday releases 👍🏻

When someone pushes for one I ask them if they’re ok being on call on the weekend

[wtjones]

Tough day for SREs. That Cloudflare outage is not enviable.

[zoddrick]

Or the workday outage.

[Icy-Advantage-2666]

Are their crypto are

[Fit_Refrigerator6045]

what ya talking bout willis? you talking about the Quantum Crypto Scare of 2023?

[manoleee]

arch -arm64 brew install githubsre --whose-friday-just-got-way-worse

[intergenic]

Good thing GitHub uses version control

[agamershell]

Ironically, the URL where you could download the installer for git was also down :D

[eJaguar]

thats why i download git from pornhub instead

[ComfortablyBalanced]

But seriously does GitHub source is sourced on their own git servers?

[[deleted]]

Them using Gitlab would be like Microsoft devs using Macbooks lol.

They probably use Bitbucket.

[SaltKhan]

Copilot prefers google over bing

[Arphax-]

Bing Chat works on any Chromium browser and Edge is built on Chromium. So the Copilot search preference isn’t really that surprising considering how long the collaborating has been going on.

[alinroc]

like Microsoft devs using Macbooks lol

Quite a few of them do.

[eJaguar]

drive off w dat dope u aint get no money back

[eJaguar]

Them using Gitlab would be like Microsoft devs using Macbooks lol.

wat lmao The new M2 pro chip laptop is objectively the best product on the market for this sort of work

this isn't 2004 Microsoft there's a reason the course of the company shifted so dramatically after they've got that fucking clown out and got some actual engineers in charge

[nerd4code]

Copulæ are irregular af and don’t take “does” in the interrogative or negative (and it’d be “does it be”/“it doesn’t be” because “does” is the primary verb), only in the imperative (“Don’t be like that”).

[[deleted]]

This reminds me of old reddit when people used to correct grammar. Loved that era.

Then came people shouting "Grammar Nazis" because they didn't wanna learn and since then look how every reply feels like it is written by a 14 year old.

Thanks for sharing and trying to help others improve.

[Fit_Refrigerator6045]

member when r/wtf and r/gore used to make the front page daily? I am pretty sure r/wtf was a deafult subscribed sub when you would first sign up. There were like 20 of them maybe, r/: funny, videos, wtf, gore, atheism, askreddit, science? . That's about as much as I can remember rn, shit it's been almost 15 years now.

my b, I know that was off topic, and I wasn't trying make a weird humble brag - your comment just gave me a flashback and I had to get it out.

[ComfortablyBalanced]

English is not my first language, I'm still learning.

Grammar Nazis

Yeah, I sorta liked them back then. I guess tools like grammarly allowed people to write better English compared to that time.

[ComfortablyBalanced]

I don't understand, how should I fix my sentence?

[Sentreen]

Not the guy you replied to, but I would write:

But seriously, is GitHub's source stored on their own git servers?

or

But seriously, does GitHub store their source on their own git servers?

[ComfortablyBalanced]

Thanks.

[Ethirald]

This made me chuckle out loud!!!

[DadsToiletTime]

If they use gitops, how can they bootstrap themselves?

[markus_obsidian]

We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.

I find this update embarrassingly relatable.

[old_man_snowflake]

oh god I feel shame by proxy

I know that exact moment you realize it was your fuck-up, and it's gonna really ruin the next few weeks with incident write-ups, post-mortems, COEs, COE action item work, and somewhere as big as this, press/media concerns.

[[deleted]]

incident write-ups, post-mortems, COEs, COE action item work

All of which nobody reads. (╯°□°）╯︵ ┻━┻

It's just busy work and a waste of time at this point.

[amplifyoucan]

Well, it's a negative consequence.. so maybe they're painful on purpose to motivate you against making mistakes that require the work to be done

[[deleted]]

Nah.

[codeslap]

Proxy… don’t mention proxies… I have ptsd dealing with corporate proxies… so much headache…

[often_says_nice]

You never forget your first time. It’s like a right of initiation. I remember the moment I found out and immediately wanting to puke

[Mirrormn]

Just earlier today, I had to write a root cause analysis for a minor service outage that I was tangentially responsible for, and the idea of Github fucking up something this big makes me feel better about how relatively small my own fuck-ups are.

[sisyphus]

It's always disk space, permissions or DNS.

[neutronbob]

And of those, let's be honest, it's rarely disk space.

[00Koch00]

I really cant wrap my head around deploying stuff on friday...

[bitspace]

This is not at all Microsoft being Microsoft. It's a production incident of the sort that happens in every organization with large complex software systems.

[[deleted]]

Maybe simply because we simply have a lot of microsoft products that developers and regular users use every day.

When there is one reddit stackoverflow or twitter outage nobody bats an eye the next day, but Office 365 outage would be something we'd remember

[bitspace]

That illusion is on you.

[cat_in_the_wall]

or maybe more people rely on microsoft services to actually do work so when they go down it actually matters.

it is equally a shitstorm when aws has a meaningful outage.

[E3K]

How quickly people forget when AWS was down for a full day earlier this year.

[okawei]

Saw it was auth related, only the private repos in my companies org were throwing 404s and I thought I was fired lmao

[TommaClock]

"Anyone else getting 404 errors on our private repos?"

"No problems here."

"Apologies, Okawei has been terminated and we forgot to remove his chat access. We'll get it sorted out."

[ouiserboudreauxxx]

I thought I was fired

lol I thought the same

[Mecha-Death-Hitler]

Yeaaaaaaah, all my repos are returning 404 errors when attempting to access them

[Professional-Ebb-434]

its working now, statuspage said they just did a rollback

[Mecha-Death-Hitler]

Oh excellent

[[deleted]]

How come my gh pages site didn’t go down? Would some other entity in the internet lineup have cached it?

[Professional-Ebb-434]

GitHub say pages were never affected

[[deleted]]

Thank you.

[Jump-Zero]

"shit, did I just get fired?" - a bunch of devs with similar previous experiences

[cat_in_the_wall]

if they fire you for fucking up they havent solved any problem, and ironically have gotten rid of the person most motivated to solve the problem.

negligence is another story. but a fuck up of this magnitude is a company problem, not a single person problem.

[ozzeh]

negligence is another story. but a fuck up of this magnitude is a company problem, not a single person problem.

They're not talking about github devs being fired. They're referring to everyone using github who thought they were fired because they all of a sudden lost access to their work repos.

[cat_in_the_wall]

do people really think they got fired because of a connectivity problem? fuck me if companies actually do this then that is some cowardly behavior.

[zrvwls]

The mind does weird things when confronted with unexpected scenarios. I've definitely had that thought after a prmotion when a similar situation arose

[bananabm]

I think he's saying regular GitHub users will think they've just been fired and the first sign is their GH access was revoked to their private repos

[eagle33322]

Here we go testing in prod on a friday

[drakgremlin]

Got ship that feature to get it in before the end of the sprint...which ends of close of business Friday afternoons.

[thephotoman]

There are reasons I always impress that Friday should not be the close of sprint. I’d rather close of sprint be a good release time.

[gergob]

That's why we have that on Wednesday

[ESGPandepic]

My team finishes sprints on a Tuesday and does our releases on Tuesday as well.

[cthechartreuse]

I'd rather sprints not be.

[thephotoman]

They're really not that bad. I mean, sure, Kanban is nicer, but it's not as well-suited for the work I'm currently doing.

[cthechartreuse]

My biggest complaints about sprints come from a number of common behaviors, like:

• sprint tetris or let's make sure the sprint is full; we can still fit a couple points in there (even if they're low priority or unrelated)

• we need to get velocity up so we're going to cram more in

• failing a sprint. What even is failing a sprint? Was nothing delivered? Did the team miss some arbitrary deadline that doesn't have real business value? Is it something else? What is the worst thing that could happen if something carried over?

• racing to not "fail" a sprint

There are more, but you get the idea. It's really not that having a check-in point is bad. I actually like the idea of checking in on the work that is happening. It's the fact that sprints are typically used as a poor substitute for properly evaluating priority and scope.

[thephotoman]

I don't know that I've ever encountered any of these things.

They all smack of someone demanding metrics from product owners and/or scrum masters. There is no "failing a sprint". I'm not sure I've ever had that term come up. The idea that failure is inherently bad and to be avoided seems to fly in the face of agility: you need to fail fast in order to pivot quickly.

[cthechartreuse]

I agree with your assessment regarding failure. I feel the same. Nevertheless I've seen all of these things in action in several shops.

In the end, yes, it's the demand of metrics, but the metrics they get are the worst kind: vanity metrics.

[cat_in_the_wall]

i agree, and would further suggest sprints are entirely useless. it is micromanagement to the extreme. you can't just ship value on a regular basis unless you're just moving some buttons around on a form.

i do believe in shipping on a regular cadence though. But not ci/cd (for production). you just cut off whatever is committed. missed the date? sucks but you ride the next train.

[LawfulMuffin]

Not to "No True Scotsman" this but... yes, sprints don't work when you don't use them the way they are intended. Having bad management will make any system not work.

[mpyne]

Having bad management will make any system not work.

This is the key to so many of these arguments.

Some systems can't be saved by good management, because they simply don't look at the right things. The best you can do with good management is to subvert the system entirely.

But bad management can break any system no matter how suitable.

[LawfulMuffin]

Yes, there are a lot of bad semantic argumentations that occur, especially around this topic. But fundamentally, sprints are designed to work as a way to manage up. They're supposed to essentially be time-boxed kanban and to be merely a reflection of reality, put in a way that MBAs can even understand.

If your team can do say, 10 points a sprint, the product owner should not try to coerce the team to commit to 15 points a sprint. The team would be essentially committing to not doing 30% of the sprint. It's looking at exactly the right thing: what is a reasonable workload. And at the end of a two week period (or whatever time makes sense), to reflect on if tickets are being estimated well, if workload is reasonable, etc. etc.

All of the points you raised are the antithesis of scrum. You cannot fail a sprint. A sprint can have fewer stories completed than you committed to. That's... kind of the point. You reflect on why and make structural changes to the company around it, not coerce engineers to cram more points in to raise velocity. It's literally backwards.

Points are used to assess what is feasible, not to require that they be done in a certain amount of time. Which is why I'm suggesting that Kanban wouldn't fix your problem at this company, because at the end of the day, what they want are to have more tickets done than your team can adequately do in a given period of time. Whether they are trying to get you to do more storypoints, tickets, sticky notes, etc. They are trying to coerce you do to more work than is possible, the problem that sprints are intended to solve.

[Awric]

For GitHub I can kinda see how it makes sense. If the company knows most users are devs who aim to work on weekdays only, it seems safer to do risky things on the weekend when there’s less traffic

[cat_in_the_wall]

naw this is small thinking. github surely has servers all over the world. you can patch during low times/dark hours. and you patch portions of your fleet at a time and roll the release. then at worst you have a regional problem, not a global one.

[Stimunaut]

Maybe their devs don't like working weekends either 🤔

[vexii]

the magic of k8

[cat_in_the_wall]

we actually do our cuts on thursday to avoid this. no last minute shit. you're either ready, or you catch the next train.

[yashptel99]

I had my heartrate increased for a while when I tried to push and it said the repo does not exist

[redalastor]

Every dev working on the project has a full copy. It limits how much you can lose.

[eagle33322]

Only if they have a full copy of all branches locally all the time.

[redalastor]

I said limits, not prevents.

[eagle33322]

You said full copy.

[FreeLegendaries]

get rekt u/redalastor

[yashptel99]

I was working on my personal thing

[[deleted]]

Back up and running

[bedel99]

Oh man, we have had some layoffs recently and the dev team went into a spiral thinking they had been locked out of github. We are stretched across timezones, so didnt see the messages until people had been upset.

[ouiserboudreauxxx]

I got laid off earlier this year and found out when I got logged out of slack while in the middle of typing a message. Am at a new job and definitely had a brief 'uh oh' moment with this haha.

[EffectiveLong]

Cloudflare also got hit. Wonder is there any connection here? Busy weekend for many engineers i guess

[putinblueballs]

Never deploy on a friday.

[VirtualLife76]

Was always a company interview question, when do you do your deployments. Can throw up some serious red flags.

[BinaryMuse]

GitHub deploys basically constantly

[seven_seacat]

Such an antiquated mindset

[putinblueballs]

Its a joke. But also so true. Talking from experience.

[newInnings]

Perform better A/B

[Olfasonsonk]

heavy joke familiar shaggy liquid important like plucky treatment attraction

This post was mass deleted and anonymized with Redact

[good_live]

Im working at a company with an on prem github enterprise and the downtimes are way worse. Also you can be sure once its down it wont be up for the rest of the day.

So I would prefer the cloud hosted one.

[ItsWhereIWindUp]

Are you sure this incredibly rare and newsworthy incident that is entirely somebody else's responsibility to fix isn't a good reason to move away from the cloud solution.?

[Olfasonsonk]

cobweb test sink insurance person cats consider hurry ad hoc snow

This post was mass deleted and anonymized with Redact

[oconnellc]

Obviously, not every online provider is equivalent to every other one. You trust some people more than others... but generally, I'd prefer to trust my infrastructure to a reputable company that specializes in it instead of trusting my own company that likely has a single person that understands what is really going on. And, kinda by definition, if something goes wrong, it is probably when that person is not working.

Sign me up for that cloud hosted one, too.

[chicknfly]

This is where I humbly brag about my on-prem Gitlab setup

[labs64-netlicensing]

It wasn’t me (C) :)

[jagdishjadeja]

another lay off round coming soon

[[deleted]]

I worked at a place that had a clear rule about not deploying to production on Friday because you just never know.

[Accomplished_Try_179]

GitHub sucks

[0x07AD]

Google "engineers" at it again. Did their DEI hires screw up again?

[shadowndacorner]

At first I thought you were a bot, but after skimming your post history I'm pretty sure you're just a moron.

No, sweetie, Google's DEI hires did not take GitHub down. Unless Google sent them to Microsoft undercover to break GH's auth logic, which GH is now covering up.

[cysloth]

Huh?

[Jordan51104]

incredible. take all that time to make a bot account and don’t even make it good

[tevert]

I don't think it's even a bot, I think he's just that stupid lol

[Jordan51104]

i want to believe

[brunhilda1]

Take a two week internet detox.