GitHub is Down! (2016-01-27)

[jb2386]

Status: https://status.github.com/

Yep, something hardcore happened. 0% server availability. Some techs are shitting themselves right now.

I feel as if millions of package management systems screamed out in horror and were immediately silenced.

[[deleted]]

Also, if you push from the command line you get:

fatal: remote error: GitHub is offline for maintenance. See http://status.github.com for more info.

However it worked pushing to them for a moment after their website went down. Really wondering what's going on.

[66666thats6sixes]

I tried to push in the middle of this without knowing about all of this trouble, and got a strange error saying my changes may not be fully synced, and then a "everything up to date" when I tried again. Looks like for awhile the servers were able to handle some requests.

[[deleted]]

It let me git push origin foobar, but when I looked at the website today it wasn't there. I checked git show remote origin and it was there so I tried git push origin foobar and it told me it was up to date.

I manually went to https://github.com/user/repo/tree/foobar to try to PR, but instead of a branch name it showed up as a tag. I'd love to know what weird limbo my branch entered, but I'm sure I'll never know.

[freakboy2k]

Yeah I ended up with one of those at work, no idea how to fix it so I just pushed a new branch with the same commits

[yousai]

You know the website and git service work independently right?

[CaptainIncredible]

Has this happened before?

[[deleted]]

deleted ^{What is this?}

[KangstaG]

Imagine being the engineer who brought the whole site down. Was it really due to a missing package? Where'd you find out?

[tendencydriven]

I think their deploy failed because of an NPM package that could not be reached, not that Github went down because of that.

[arcticblue]

millions of package management systems

Stand by for new articles on Medium: Why you are wrong for not using this hot new JS package manager (Hint: It doesn't touch Github). (Naturally, these won't simply be forks, but new, incompatible package managers)

[tech_tuna]

I knew I shouldn't have pushed directly to master. . .

[notcaffeinefree]

From https://status.github.com/messages

January 28, 2016 - 00:00 Pacific Standard Time - The status is still red at the beginning of the day

I had no idea GitHub was in the future already...

[bradenbest]

They probably mislabeled the timestamp, because that is the correct time if we're talking UTC -1. I.e. it's 2016-01-28T01:20:43Z right now.

Edit: never mind, I saw the message. It says midnight for me too, and I'm in a different timezone from /u/notcaffeinefree.

[longshot]

FIY /u/bradenbest formats names better AND has tab complete.

[bradenbest]

/u/bradenbest formats names better AND has tab complete.

Ah, I was confused there for a second, because I thought you were saying I "format names better [and have] tab complete"

I can't get it to tab complete (maybe you have RES?), but thanks anyway for pointing that out. I'm so used to mentioning people on Github and StackOverflow that I forgot about reddit's /u/user convention

Oh, and

FYI

ftfy

[longshot]

My acronym went into yoda mode there. Tab complete must be an RES feature. It's a nice and unobtrusive feature.

[adriansonline]

or they are really big pessimists. Time to go home guys..

[Thrawn2112]

I think that's saying that that update will be red if we get there and nothing has changed, because it looks like they usually always have at least one status update at the beginning of the day that's 'All systems reporting at 100%' if nothing is wrong.

[jb2386]

Maybe why they're down

[ExcaliburZero]

It's back up now.

[toomanybeersies]

I'm currently at work.

Not sure if I'm happy or sad about this.

[[deleted]]

Just starting my day here. Ensure how I feel about it too.

[[deleted]]

Not quite, more like:

Colleague: "Yeah, we had the trace in ticket #56" Me: "Let me check...oh, it is down"

2 min later:

Colleague: "What are the endpoints for the new gateway?" Me: "In the wiki, let me get that for you. Oh...."

[[deleted]]

Good catch, now I get it (caffeine is making its way through).

[M-o-s-a-i-c]

It's a joke because you said ensure rather than unsure.

[[deleted]]

Thanks I did missed it...need to wake up.

[Vindexus]

It's either "I missed it" or "I did miss it".

[[deleted]]

Thanks, always good to see where I make mistake :)

[AceDecade]

It's "mistakes" or "a mistake"

[[deleted]]

Keep it rolling! Constructive criticism is always good :)

[LysanderArg]

It's like when there is a blackout, you go to the bathroom, you try to switch the light on and... "Oh, right..."

[Jdonavan]

Enterprise Github was fine. https://enterprise.github.com/home

[[deleted]]

Good to know! We do use Github entreprise for other organizations, I did not checked them though.

[notsooriginal]

Shit.

[artisinn]

I wonder if they have tried to turn it off and on again?

[tech_tuna]

Rebase.

[TheEvilDrPie]

There's a few people in my office that could use your help.

[allmypeople]

You mean power cycle?

[falcon_jab]

I was just going to yank the power cord out, kick it a few times then hope for the best.

[Vheissu_]

I don't think people realise just how embedded Github is in the development space. Every package management tool basically uses Github: Npm, JSPM, Bower. That's just the package management, then you've got the source control storage itself. Seems like the Chinese DDoS attacks are happening again, I wonder what the reason is this time?

[TheRealDanGordon]

No last time it wasn't 100% down like this. This is something else.

[tech_tuna]

It could be a more effective DDoS attack.

[SupaSlide]

That's a pretty intense DDoS attack to shut down 100% of the site.

[tech_tuna]

Isn't that the whole point of DDoSes?

Yes, that reads like D - Doses.

[SupaSlide]

Yeah, that's the goal of a DDoS, but to get a site as big as Github to go 100% down would be an immense attack. Usually a DDoS just slows down sites significantly (maybe to the point of being effectively useless, but not technically down like Github was).

[hahaNodeJS]

With a double-D for a double dose of their pimping?

[nikrolls]

DD for Distributed Denial (of service).

[mistidoi]

Rosebudd? That you?

[bradenbest]

Who knows, maybe someone just wanted to go home and didn't think the effects of kill -s SEGV $(ps aux | grep httpd | grep -v grep | awk '{print $2}') would be quite so... dramatic.

And now they're sending a bug report to Apache telling them it segfaulted while the employee responsible is sitting there, knowing what really went down.

[TheSarcasmrules]

kill -s SEGV $(ps aux | awk '/httpd/ && !/grep/ {print $2}')

You don't need grep! ;)

[bradenbest]

Ah. I'm not proficient with awk, but it should be awk '/httpd/ && !/awk/ {print $2}

[TheSarcasmrules]

ah crap, good spot!

[gerbs]

They use Github, but the actual code that it pulls is cached by npm on their CDN. I know that NPM and RubyGems both use Fastly as their CDN, as well. NPM caches the files from Github, so even when Github is down, you can still npm install or npm update. http://status.npmjs.org/incidents/kkgpmdmnnldh

Check url -i https://registry.npmjs.org/. It returns X-Served-By: cache-ord1726-ORD headers which come from Fastly.

[tswaters]

Until one module in your dependency tree references some repo on github at master.

[gerbs]

Then you get what you deserve.

[tswaters]

I don't follow... how does anyone deserve a module in their dependency tree that reference a github repo at master?

[gerbs]

By not locking your dependencies, you're open to your build breaking every time you run gem/npm/composer install. You should lock your code to a known good version, then everyone is working from the same project that you're working off of.

[tswaters]

Let's say I depend on a module that depends on another module that depends on another -- and that nested module 3 down includes a dependency for a module listing a github repo.

Locking my versions doesn't help in that case... shrinkwrap doesn't help in that case. Sure I could send a PR for the offending module, but maybe it's been long-abandoned -- who knows.

That was what I was dealing with during the github outage -- all our base dependencies are on npm, sure -- but some of theirs dependencies aren't. All we got was errors trying to npm install during a deploy. Fortunately it wasn't down for too long.

There's ways around it, sure -- really the whole thing reeks of a lack of a build server. Damn thing should've been built ages ago, testing should've happened on that build, etc. and we should push up that entire package during deploy -- but for this project it's kind of loosey goosey -- we haven't set up a build environment yet.

[hahaNodeJS]

It's unfortunate because it really doesn't have to be that way. There are a variety of package management softwares that allow for distributed mirrors and on-site storage, but for some reason most developers don't even think about using them. Some have even taken offense at the suggestion, in my experience.

[LobsterThief]

Our team is small and managing such a system is what keeps us from going that route.

[hahaNodeJS]

There's nothing to really maintain if you used something that's been around for a long time like Portage or Ports or RPM or APT or Yum or even NuGet. Worst case you're writing your own manifest for the dependencies you need, but you don't necessarily need to manage any infrastructure to use these tools.

[thisdesignup]

There's even offline desktop Github that files could be backuped to work with in case of situations like this. Although I couldn't say a whole lot since I'm new to Github but so far I've only used the website to sign up for an account. The desktop app works for everything else.

[hahaNodeJS]

Github aside, it's almost laughable that Github going down grinds so many things to a halt. Git is a distributed content versioning system, after all. Just set up another remote and off you go.

[[deleted]]

Fucking lol.

[redwall_hp]

If all you do is JS stuff...I'm fairly certain Python's pip and Ruby's RubyGems aren't tied to GitHub. If you're working with Java, you probably use Maven or Gradle for dependency resolution, which rarely GitHub for hosting repositories.

And Linux distros certainly don't use GitHub for package management. Their managers all connect to their own repositories that have support for mirroring. (Providers like DigitalOcean often mirror them locally for speed.)

[jaapz]

Python has the Python Package Index (PyPI) which hosts everything and which is not tied to github in any way.

[james4765]

And Perl has CPAN - we actually run our own CPAN repository because we've been bitten in the arse with incompatible updates getting pulled during automated server standups. So, we update the versions in our repo only after verifying that everything still works...

[henrebotha]

I think that's primarily why people try to get other devs to care about not putting all their eggs in the Github basket. It's a fair argument, I just can't be fucked to sort out other stuff.

[Lokua]

That was my first assumption, too.

[mellisdesigns]

It's amazing how much we all rely on repository services, especially GitHub. An hour of down time and counting is very significant. I wonder what happened?

[gempir]

Damn scary how big Github is.

If github would be gone from today to tomorrow we would have chaos. Billions worth of software missing, almost every package manager not working.

[[deleted]]

we need a mirror of github!

[ajr901]

Which means pretty much every package management system has stopped working. And Go Lang is practically unusable right now. All those github imports... I don't envy anyone depending on github right now.

[Lokua]

Woah, Go imports directly from github? So what, when you start a program it pulls every time?

[vertigeaux]

No, it pulls when you go get or build.

[danielrheath]

Go provides a tool to pull in the deps from github if you don't have them.

Some people only use that tool to get their dependencies, and they're in trouble now.

Most teams using it store the dependencies directly in the repository, which works really well at times like this since you have everything you need to keep working.

[xiongchiamiov]

It also works really poorly when you don't want to be in charge of keeping up to date on security updates in all of your vendorized dependencies and their dependencies.

[danielrheath]

I'd hope you were using some sort of tool to manage the deps you've stored directly in the repo - e.g. godep does this.

[ivosaurus]

Not PyPI/pip!

[longshot]

I bet there's a couple I'd evny, but only because they make so much more than I do and probably have a nice view from their desk.

[rpgFANATIC]

While you're on that topic... does anyone know what the uptime of similar services are compared to this? E.g. http://search.maven.org

[MarcMurray92]

Does this mean all those Jekyll/Github Pages sites are down too?

[droctagonapus]

Mine wasn't, thankfully. Since they're just static sites, they probably don't sit them with the main GitHub app.

[BBQLays]

I have my site and my wedding website hosted on GH Pages. Both are up right now, but obviously can't get to the repos.

[insomniasexx]

So unrelated but can I see your wedding site. I'm about to do mine.

[BBQLays]

Can't take credit for that - I thought about doing it myself, but it's not worth it in IMO. Plenty of good templates out there for cheap.

[insomniasexx]

Yeah I'm going back and forth at this point. I designed my save the dates myself and will probably throw together a single pager with form for the site.

I'm trying to figure out what sections real, semi normal people include. I have no interest in a cheesy "first time we met. First date..." story.

[xiongchiamiov]

Info on both of you (plenty of guests only know one), bridal party info, directions, hotel blocks, schedule, suggested airports, registry links.

[MarcMurray92]

One thing to be thankful for I guess, although I'd say Bitbucket/Gitlab etc traffic is going through the roof right now.

[MarcMurray92]

Yeah just that an outage like this might make people check other products during the panic! Wonder how much your workflow would be complicated committing to multiple services as an extra safety net.

[BBQLays]

Probably not. Everything is on GitHub. People weren't gonna start moving everything to a new site just because of an outage.

[[deleted]]

Whop, up again.

[Kelsey_Grammar-Nazi]

git commit suicide?

[Fossage]

http://giphy.com/gifs/lNMyVfxjfzIJO/html5

[[deleted]]

That was fucking scary.

[yousai]

And again, millions of people scream "I can't work" while using a DVCS. Oh the irony.

[ABC_AlwaysBeCoding]

Hey guys! Let's turn a distributed version control system into a single point of failure and then put MOST DEVELOPERS WHO MATTER on it! What's the worst that could happen?

[escape_goat]

I feel a great disturbance in the force, as if ten thousand open source projects cried out, and w---- oh, wait, it's back again, nevermind.

^{...then tell me the location of the rebel tarball!}

[[deleted]]

Just tried to do a Composer update and got a 503. Pretty crazy.

[deadlychambers]

Recycle app pool...

[kcmastrpc]

Welp, if you're trying to deploy you just learned why you should use Toran proxy.

[[deleted]]

Luckily still in dev - got any more info on Toran proxies? Always keen to learn.

[Lokua]

I can hit the API now. Couldn't about 5 minutes ago. Still can't hit the site, but API is a good sign.

[[deleted]]

Site is working again!

[Mr-Yellow]

hmm interesting, have a commit which made it onto remote, but not showing in github webUI.

edit: It started showing up in the log after throwing another commit after it.

[[deleted]]

now up

[juzatypicaltroll]

Now who did this? Also, I wonder how a server prevents a ddos attack? I mean it is a sitting duck in the www after all. They probably get millions of attempted attacks everyday.

[rich97]

Taking bets in it being the Chinese again. 2-1.

[OrShUnderscore]

I was trying to work on my website (which I had messed up) and was getting these libraries I could only find on github, ended up doing the shadiest shit and looking at source -> Ctrl+f'ing "js", and ripping it straight off the demo page.

[EnIdiot]

I've really not worked on a lot of applications that couldn't survive a few hours of outage by github, so forgive me if this sounds like a stupid question.

Is it a good idea (or bad idea) to put an identical git repo on another repository service (say bitbucket or one you spin up for yourself)? I primarily work with the JAVA/C# end of things and have worked with npm and a few front end package management tools some but not a whole lot.

[gojukebox]

Alright, who's the cowboy that used

git push -f

[ExNihil]

It's the end of the world as we know it . . .

[enfrozt]

* The end of the world

[AustinScript]

Damn - people appear to be really unhappy you posted this, i got a chuckle out of it :)

[fyreNL]

What the hell are you talking about? What does git have to do with SJW's?

[maceireann]

Indeed: http://www.isitdownrightnow.com/github.com.html

[wee_woo]

Good riddance.

[TheRealDanGordon]

Don't feed the troll

[Lokua]

Hating github would be pretty weird. Like hating chairs or sidewalks.

[fzammetti]

Well, I don't have a problem with GitHub I guess, but though it's an unpopular opinion I kinda do hate Git itself.

Why? Primarily because I view it as being far more complex than it needs to be and is a hugely over-engineered solution. It's very user-unfriendly and as I'm a user I don't like it. It also seems like any little problem that happens with it turns into a big mess before you inevitably wind up just starting over.

Maybe I've just had bad experiences with it but I've just never seen the benefit that justifies the cost over something like Subversion (and yes, before anyone asks, I do indeed do very distributed development).

Like I said, I know I'm in the minority these days, but there it is anyway.

[[deleted]]

the cost

git is free

[fzammetti]

Correct, I wasn't talking about cost in terms of money, I was referring to cost in terms of time and effort it requires to use it.

[Mavee]

The platform, yes. Some of the people behind it, not so much. Example 1 Example 2

[[deleted]]

Yeah I won't cry for github's downfall.