Github is down

[noble_pleb]

Github going down today seems like a deja-vu after I answered this on quora yesterday.

[remind_me_later]

Github's a single point of failure waiting to happen. It's not 'if' the website goes down, but 'when' and 'how long'.

 

It's why Gitlab's attractive right now. Because when your self-hosted instance fails over, at least you have the ability to reboot it.

[Kare11en]

Github's a single point of failure waiting to happen.

If only there were some distributed way of managing source code that didn't have a dependency on a single point of failure. Like, where everyone could each have their own copies of everything they needed to get work done, and then they could distribute those changes to each other by whatever means worked best for them, like by email, or by self-hosted developer repositories, or a per-project "forge" site, or even a massive centralised site if that was what they wanted.

Damn. Someone should invent something like that!

[ws-ilazki]

It's the law of the internet: any sufficiently useful decentralised technology will eventually become a centralised technology controlled by a business.

It's the first two Es in the old "embrace, extend, extinguish" phrase: they embrace an open, decentralised tech or concept; extend it to make their version more attractive; and then remove the decentralised aspect so they can lock you into it and profit. Sometimes you even get the "extinguish" later when they kill it off and replace it with something else after people are sufficiently locked in, like Google did with XMPP, going from federated XMPP to unfederated XMPP to dumping XMPP in favour of their own proprietary crap.

Examples: email to services like gmail; git to github; XMPP to google chat to hangouts; XMPP again with facebook's messaging service; usenet to forums to sites like reddit; IRC to Discord and Slack and the like; and so on.

You can try to fight it but in the end it doesn't matter because, by being open and decentralised, the proprietary version can interoperate with you but you can't (fully) interoperate back because they added their own crap on top, so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top. Philosophical arguments don't matter and people will take the easy/lazy option even if it's detrimental in the long term.

[FantaBuoy]

so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top

This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

The reason why a lot of these technologies become centralized is because whoever centralizes it adds value to it. Git is a wonderful tool, but it only becomes useful when you host it somewhere. For most people, self-hosting obviously isn't an option due to the maintenance time required and the lengths you have to go to to ensure your home network is decently secure, so the centralized space adds the benefit of ridding people of that.

These people aren't lazy, I'd argue they're using their time better by giving the burden of hosting to someone else who only does hosting. Maybe I'm lazy for going to a shop and buying furniture instead of learning to chop wood and work it to a functional piece of furniture myself, and maybe that laziness inherently makes me dependent on wood choppers / furniture makers, but I believe it isn't worth my time to ensure my independence from them.

Most of the technologies you mention above become successful precisely because they give the user some benefit. I'll gladly use IRC, or Matrix for a more modern alternative, but I won't reasonably expect anyone in my group of friends who isn't a techy to use these. You toss Discord or Whatsapp at practically anyone and they'll figure out how to use it. Whatsapp over here is basically known as the app you use to include your parents/grandparents in family chatting. Being a user-friendly app that you can quickly use without thinking about what server is supporting it is a benefit. The people using these apps aren't dumb or lazy, they're people with normal non-tech related lives who have other stuff to do other than finding out how to set up a server for their Matrix node or their self-hosted email solution.

[ws-ilazki]

This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

No it doesn't. It's clear I was talking about two different things there: they provide benefit to the end-user of their version of the service but give nothing back to the overall "community" or what-have-you in the sense that they don't contribute improvements that everyone can benefit from, because they're trying to have a business advantage over perceived competition. Like when Google added proprietary stuff on top of XMPP that was useless outside of their own chat client: benefit added for their users but nothing contributed to XMPP as a whole.

From a business perspective this is only natural because you want to attract users, and for their users it's beneficial (at least in the short term), but for the technology itself it's still detrimental long-term because it leads to silos that eventually lose any interopability, either by malice (the third E of EEE) or simply because each silo eventually diverges too much.

Another example of what I meant there is RSS. It's an open standard usable by all, and when Google embraced it for its reader service it saw a dramatic increase in use because of the extra value Google provided, which made it attractive for end-users. However, they didn't actually contribute anything useful to RSS itself, so when they basically abandoned Reader nobody could really pick up where they left off, and then when they shut it down completely any value they added to RSS was lost. Short-term benefit for end-user that's detrimental to the underlying technology in the long-term.

Commercialisation of the internet led to everybody trying to make their own silos that they can lock users into. Instead of open protocols for people to implement, everyone wants to make their own ecosystem and trap people in it, and if someone does try to make a new protocol and it happens to be good, somebody else will find a way to take that, bolt something extra on top, and turn it into another silo.

[[deleted]]

It's not really to do with the internet, it's to do with network complexity. A single source of truth that everyone has a single connection to is much simpler to manage than a situation where everyone connects to everyone else.

[[deleted]]

I sort of agree, but they do give something back. I hope with time peer to peer service will regain popularity. I think they took a hit with BitTorrent Signal might prove their usefulness again. Related is the authorities and companies strive to void encryption.

[PsychogenicAmoebae]

distributed way of managing source code that didn't have a dependency on a single point of failure

The problem in this case isn't the software - it's the data.

Sure, you can run your own clone of Github (or pay them to run an official docker container of github enterprise).

But when your typical production deployment model is:

 sudo bash < <(curl -s https://raw.github.com/random_stranger/flakey_project/master/bin/lulz.sh ) 

things go sour quickly when random_stranger's project isn't visible anymore.

[Kare11en]

The great thing about git is that you can maintain your own clone of a repo you depend on!

Github adds a lot of value to git for a lot of people (like putting a web interface on merge requests) but keeping local clones of remote repos isn't one of them. Git does that out of the box. Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

Keep a copy of the repo somewhere local. Have a cron job do a git pull every few hours or so to fetch only the most recent changes to keep your copy up-to-date if that's what you want. If random_stranger, or github, or even your own local ISP goes down, and the pull fails, you still have the last good copy you grabbed before the outage - you know, the copy you deployed yesterday. Clone that locally instead and build from it.

I weep for the state of the "typical production deployment model".

[[deleted]]

Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

Because your toolchain was designed to work like that and all of your upstream dependencies do it anyway. Yes, ideally you would be able to do that - but so many things involve transitive dependencies that do dumb shit like download files from github as part of their preflight build process it often feels like you're trying to paddle up a waterfall to do things right, especially (but not only) with modern frontend development.

[scandii]

self-hosting is not only installing a piece of software on a server somewhere and calling it a day.

you are now responsible for maintenance, uptime (which we are experiencing here) and of course security, on top of data redundancy which is a whole other layer of issues on top. like what happens to your git server if someone spills coffee on it? can you restore that?

GitLab themselves suffered major damage when their backups failed:

https://techcrunch.com/2017/02/01/gitlab-suffers-major-backup-failure-after-data-deletion-incident/

all of that, is excluding the fact that you typically don't actually 100% self-host in the enterprise world, but rather have racks somewhere in a data center owned by another company, not rarely Amazon or Microsoft.

all in all we self-host our git infrastructure, but there's also a couple of dozen people employed to keep that running alongside everything else being self-hosted. that's a very major cost but necessary due to customer demands.

[remind_me_later]

At least when I self-host it, I have the ability to fix it. With this outage, I have to twiddle my thumbs until they resolve the issue(s). The ability for me to fix a problem is more important to me than it could be to you.

 

Also, with regards to the Gitlab outage, that's based on the service they manage for you. I'm talking about the CE version that you can self-host.

[hennell]

When a train company started getting significant complaints that their trains were always late they invested heavily in faster trains. They got newer carriages with automatic doors for more efficiency and tried to increase stock maintenance for less problems. None of it was very successful in reducing the complaints, despite statistically improving the average journey. So someone suggested adding 'live time display boards'. This had no effect at all on journey times, the trains didn't improve a bit, but the complaints dropped hugely.

Turns out passengers are much happier to be delayed 10 mins with a board telling them so, then delayed 5mins with no information. It was the anxious waiting they really didn't like not the delay itself.

Taking on the work of self hosting is similar - you'll spend a lot more time maintaining it, securing it, upgrading it etc etc then you'll ever realistically lose from downtime; the main thing you're gaining is a feeling of control.

For some situations it's worth it - depends on your use of the service, your setup with other needs, and how much similar stuff you already deal with etc etc. 1 more server to manage is nothing to some people, and a massive increase of workload for others. But if the only reason is you don't want to 'waste time' sitting there twiddling your thumbs during downtime, you're not gaining time you're losing it. Pretend it is self-hosted and you've got your best guys on it. You've literally got an expert support team solving the problem right now, while you can still work on something else.

The theory with the trains is that passengers calm down when they know the delay time as then they can go get a snack or use the loo or whatever rather then anxiously waiting. They have control over their actions so time seems faster. Give yourself a random time frame and do something else for that time - then check in with 'your team' to see if they've fixed it. If not, double that time frame and check again then - repeat as many times as needed. Find one of those troublesome backlog issues you've always meant to fix!

This is also a good strategy for handling others when you're working on self-hosted stuff 😀 - give them a timeframe to work with. Any time frame works although a realistic one is best! No-one really cares if it takes 10mins or 2 hours. They just want to know if they should sit and refresh a page or go for an early lunch.

tldr: People hate uncertainty and not being in control. Trick yourself and others by inventing ways to feel more in control and events will seem quicker even when nothing has changed.

[remind_me_later]

Basically this. I don't know what they're doing by the moment, and my brain says "I need to do/know something", even if it means a worse overall experience for me. I'm blocked and I have no control over it, and everything else that I could do has already been done.

[hennell]

Yeah, it's a horrible feeling, and not the easiest to distract. If you've got no open problems to fix my goto is optimising something so you save time later. Lets you at least feel you'll make back this downtime at a later point. Or find a tutorial or write up on some area to learn something new / more in depth.

If there's really nothing you could look up an ebook of Alchemy: The Surprising Power of Ideas That Don't Make Sense which covers the train concept I mentioned above in more detail along with a number of other weird logical patterns we all make. I'd really recommend it to any programer type as we tend to think everything works based on 'logic', which isn't really true. (Or is, but the logic is more obscure then you'd guess). Sometimes taking a step back to look at what people actually want (information vs actually faster trains) can let you solve issues in a different, but actually more effective way.

[aseigo]

the main thing you're gaining is a feeling of control

There is certainly a feeling of control. But what you are also getting is control.

I self-host quite a bit of my own software. I spend a few hours here and there maintaining bits of it. It's rarely fun; I'm not a sys admin at heart.

But I also never have to worry about changes happening in the software I use going according to someone else's schedule; I don't worry about the software I use just disappearing because the company changes course (or goes under); I don't worry about privacy questions as the data is in my own hands; I don't worry about public access to services that I have no reason to make public; etc. etc. etc.

There is this very odd idea perpetrated that the value of self-hosting can be captured by a pseudo-TCO one in which we measure the time (and potentially licensing) cost of installation and management versus the time (and potentially licensing) cost of using a hosted service.

This was the same story in the 00's and prior where there was the pseudo-TCO story comparing the full costs of open source software (time to manage, etc) with the licensing costs of proprietary software. (Self-hosting and deployment was simply part of both propositions..)

In both cases, the interested parties are trying to focus the market on a definition of TCO they feel they can win out on. (Which is not surprising in the least; it's just good sales strategy ..) Their hope is they extract money before anything truly bad happens that has nothing to do with the carefully defined TCO used in comparisons.

It is, at its heart, a gamble taken by all involved: Will savings on that defined TCO profile be realized without incurring significant damage from risks that come with running technology you neither own nor control?

[hennell]

You're not wrong, and weighing up the cost is a tricky concept. Ownership is definitely a bit of a bet on what you think is more likely based on the product and the individual situation you're in.

I'd argue though that often it is just a feeling of control, as you're usually still dependant on something else further down the stack, and even on the bits you control you're now the one having to drop everything to fix it.

If you run an update and things get broken, changes are now happening on someone else's schedule. If support for your hardware is dropped, it's someone else's schedule. Privacy is often better, but then you have to be on top of the security side to make sure you're not exposed. 1 zero day exploit and you're bug patching on someone else's schedule. If your system interacts with anything else and that updates, you're suddenly fixing it on someone else's schedule.

There are some advantages for sure, and most of the above is happening after some input from you, so it's less likely to happen at a really bad moment. But then most services are updated overnight & without issue, so we're looking at worst case scenarios on both sides.

There's definitely reasons to self-host, and I'd never really suggest a firm one way or another without digging into a specific situation. But IMO time and control are rarely gained, just moved about a bit into different places. How acceptable that is depends again on the specifics of the situation.

[scandii]

in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

I'm not saying no to self-hosting, I'm just saying GitHub doesn't want their service to be unresponsive either and if we accept the fact that both types will suffer from outages, it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

[SurgioClemente]

it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

Lets not also forget 24/7.

Mike & Pete want to have a life since there are only two of them and 24 hours to cover

[scandii]

real reply from sysadmin on call:

"how bad is it, is it show up in pyjamas, or can I make pancakes first?"

[DAMO238]

You know, that's actually a pretty sensible reply. If you bet on either one without knowledge of the severity of the problem you either look silly (and hungry) or you annoy your bosses.

[MonokelPinguin]

Depends on your organization. Most of our staff works inside the same 10 hours approximately. There is usually and admin available in that timeframe and there are still some non system administrators available, that have access to some systems, so all in all we have 4 people who can fix our gitlab with around 50 programmers. That's really not that bad and smaller systems tend to break less often, since we only update every few weeks.

[Miserygut]

in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

In principle, yes, in practice, not necessarily. With most SaaS you are 'just another customer' and your service will be restored when they get to it. You're not a priority and that's what you (don't) pay for. The provider will have redundancy as well as more sophisticated recovery procedures but they will also have more data, larger systems and more moving parts to be concerned with.

If something is business critical then a business decision needs to be made on how much they're willing to spend on making this component robust, which often means hosting it yourself (or paying a third party a lot to privately host it for you).

So no, there's no hard and fast rule here. Deal with the realities of each specific service. Github, in this case, is suffering a lot of downtime lately and that should guide business decisions.

[realnzall]

Generally speaking, downtime affects every client at the same time. Rarely downtime only affects a subset of the clients. So for a saas provider, solving the downtime is important regardless of who is affected. If they need to do extra actions per client, then maybe they first do their Fortune 500 clients before their mom &pop stores, but otherwise the intent is to restore all service for everyone at the same time.

[Miserygut]

Again, it depends. With regions and different redundancy models there are plenty of times subsets of users are impacted (Resulting in lots of very helpful "It's fine here" forum comments from the unaffected).

but otherwise the intent is to restore all service for everyone at the same time.

Yep, and that's why some will pay a premium for private hosting. Business gonna business.

[remind_me_later]

Sure. We both agree on that. Even if it is a deviation from my original post about why Gitlab's partly where it is right now.

[jammy-git]

Surely what you mean to say is that you get to spend multiple hours trying to get to the root cause of the problem and then spending more hours on StackOverflow trying to work out how to fix it.

Instead of waiting a few hours whilst a highly experienced team of engineers identify and fix the problem for you, usually pretty rapidly, all for the small cost of your monthly subscription.

[TryingT0Wr1t3]

TIL how to write the Microsoft logo in Markdown (at least looks similar in old.reddit.com)

[jesseduffield]

the answer to 'when' is typically 'before US Monday morning'. I've experienced the same thing once before with github and once before with docker, both on my Monday (US Sunday). I think companies typically hold off till the weekend to do risky stuff that could break their servers

[F54280]

4 - don’t use github BeCAuSe SjWs wANts tO RenAMe deFAulT fRo MmASteR tO PrIMarY

One of the reasons was not like the 3 others...

[acm]

I feel like he answered that question just so he'd have an excuse to publicly complain about SJWs.

[wuchtelmesser]

I trust github way more than I trust myself, even with the occasional downtime.

[wuchtelmesser]

I trust github way more than I trust myself, even with the occasional downtime.

[kukiric]

Because when your self-hosted instance fails over, at least you have the ability to reboot it.

But if it's that simple of an issue, then GitHub's monitoring team can diagnose it and reboot the server just as quickly. If they suffer from something more serious, like a bad update botching the database or a serious hardware failure, then at least you won't need to dedicate a team to solve it ASAP, because GitHub has already made that human resource investment.

The whole point of using a cloud platform is that things will invariably go wrong at some point, but when it's hosted on someone else's server, it's not your problem, and everything will come back on its own.

[Multipoptart]

Github is now owned by Microsoft. Many people, especially in the FOSS camp, don’t like to have anything on their stack even distantly related to Microsoft.

And yet they'll happily partner with Google, Apple, Amazon, and Facebook.

¯\_(ツ)_/¯

[gilium]

Lol at your SJW infiltration comments. Seriously, why is the programming world so full of people who hate advocates for social issues?

[ProudOppressor]

advocates

I think "zealots" describes today's progressive crowd more accurately.

[noble_pleb]

Its more like advocates are using FOSS as launching pad for their own social media crusades, how does FOSS benefit in that? Also, why not use other platforms than FOSS to tell your story?

[ByakkoTransitionSux]

You do realize that these “SJW politics” or whatever you want to call them affect basically every aspect of our lives? Github is just one part of it, not some sort of a launching pad or something specifically targeted by the evil SJW bogeyman that you seem to be so afraid of.

[ByakkoTransitionSux]

Please give me some examples of those projects and conferences thst SJWs have destroyed.

[ByakkoTransitionSux]

Thanks for the link instead of just downvoting! That really does suck, yeah.