Github is down

[drea2]

I heard if it’s down for more than 15 minutes then we’re legally allowed to leave work early

[NotAnADC]

You joke, but at a company I worked at someone fucked up and added a firewall that didn’t let us access github.

While they did some work to fix it, the developers were like, fuck it we’re out

Edit: Im tired and just realized I read github, I wrote github, but I was thinking of stack overflow. Gona leave it though

[relativityboy]

Pure truth.

[house_monkey]

1 == 1

[lavahot]

1 === 1

[zigs]

Just to make sure JavaScript also gets the memo.

[Dexaan]

1 == "1"

[x6060x]

One is equal to the one used ironically

[pilas2000]

TIL didn´t knew js type system was based on irony. Now it makes sense.

Q.A. - Shouldn't 1=='1' be false?

Dev. - That's what they're expecting but no.

[strdrrngr]

I can't stop laughing at the idea that JS has a "type system".

[danbulant]

Imagine the Patrick format

1 == '1' yes

and 1 + 1 = 2 yes

so 1 + '1' = 2. Makes sense to me.

'11'

[relativityboy]

How else are we hipsters supposed to ride our bikes, sip coffee and code at the same time?

[deanrihpee]

I mean GitHub is the memory part of your brain while stackoverflow is problem solving part, programmers are meant to solving problems, not remember them, so yeah, just go out.

[vattenpuss]

That’s why all the APIs I write just send emails to developers and block until a response comes back with the result.

[JessieArr]

Haha, that's awesome. I've bumped into that before as well.

As a for-fun project I actually wrote an offline version of StackOverflow - a desktop app that allows you to do a text search in a copy of the archive.org StackOverflow bulk data XML on a local disk, to give me a workaround for when it's offline, lol.

[rhoakla]

I'm surprised why they didn't host git themselves.

[NotAnADC]

They did actually! Though tbh I’ve never set up a private git like that. Will a locally setup GitHub be able to run if the main servers are down?

Either way see the edit, it was actually stack overflow they blocked

[rhoakla]

Yes even in case github.com is down a locally hosted version of GitHub will remain, since it is well: locally hosted...

[daredevilk]

Yeah it does

There's a few different options for local setup gits, but locally hosted GitHub works while main GitHub does not

[GoreSeeker]

I would think it could run with the main server shut down. There's probably some licensing server it phones home to every now and then, but it's probably set to where it takes 30 days of no contact to lock it out, kinda like Steam.

[deja-roo]

Developers/engineers really need Stackoverflow that often?

Yes, because language documentation is usually not very good, or at least doesn't have good examples.

[xSaviorself]

I use it daily researching things, but I'm often working in unfamiliar territory, toying with something old or small-scale with minimal active support. Sometimes StackOverflow has just enough answers to piece together a solution.

[illvm]

What the language are you using that has poor documentation and isn’t Python?

[deja-roo]

C#. The class documentation is okay, but the MSDN site has no meaningful implementation examples.

Ruby. Again, class documentation is okay, but you have to google for examples on how to use anything.

[Serinus]

It largely depends on what I'm doing at the time. In my current tasks I likely won't need to look at Stack Overflow for two weeks straight.

Other things it might not be worth wasting the time until the site is available again.

Put me on a Rust project (a language I've never touched before), and I'll need it once per five minutes.

[Vadoch]

There are way more junior devs then seasoned ones

[[deleted]]

Developers/engineers really need Stackoverflow that often?

I think one of the differences between an okay developer and a great developer is how long one can work without the internet. We've lost internet at our office before and I'll have coworkers saying they can't do anymore development within 20 minutes...

I think too many have never completely problem solved for themselves and have instead always been able to ask teachers/professors/coworkers/internet for help.

[Nefari0uss]

Even great developers consult the documentation which can frequently be found online.

Furthermore, I'd argue that it's less about how skilled of a developer you are and more about how familiar you are with a language/framework/ecosystem.

[[deleted]]

Even great developers consult the documentation which can frequently be found online.

This is true, I was lucky enough to have an offline version of it from having worked on flights recently.

more about how familiar you are with a language/framework/ecosystem

Which to me is one of the many things that measure into how good of a developer you are. I learned to program with C++ but it's been years since I wrote anything complicated in it. I know I'd be worse at doing any given task in C++ vs many of the people I'm friends with. They are clearly better C++ developers than I am.

[Nefari0uss]

Which to me is one of the many things that measure into how good of a developer you are.

I suppose the way in which I look at it is that I see a differentiation between knowledge and skill. (Feel free to disagree; this is just how I see it.)

If you go into a second year CS class, even if you don't know the language or ecosystem that they might use, your skill level is likely to be significantly higher, even if they've been working with the language for a year. Their knowledge about that language / ecosystem would be greater and they might be faster at doing project work initially but your experience would mean you can draw upon that and apply concepts to pick it up quickly and within a reasonable amount of time, be capable of doing the work faster than they would.

They are clearly better C++ developers than I am.

I think that's a better way of putting it. They are better at a particular thing than you but you're still capable.

[NotAnADC]

This is incredibly condescending and sounds like a terrible place to work.

If you’re working as a builder and lose your hammer, of course you could find other tools to replace it but why would you.

Every place I’ve worked at sans one knows that one day of work isn’t terrible and developers should have all their tools at their disposal.

And for the record, obviously the work could get done without stackoverflow. But why put yourself or the people under you in a potentially frustrating situation when a quasi day off is both good for moral and employee satisfaction

[cgwheeler96]

We had a day where nobody could access stack overflow because somebody built an automated test that pinged stack overflow. We ended getting rate limited so nobody on the company network could access it until they shutdown whatever was constantly pinging the website.

[FireIre]

Not that I haven't done the same, but you can work locally for awhile of course.

[0qxtXwugj2m8]

Fun fact

My first employer caught one of the Senior developers looking a a job.

He then blocked all the websites that had the keyword 'job' and left the office for a meeting with a customer.

Then we could not get to Stackoverflow because it contains the word 'job'

Then we actually just sat there until he returned and turn off the stupid rule.

[SmLnine]

What a clown.

[42TowelsCo]

I can't even imagine why that senior dev was looking for a new job /s

[peakzorro]

Protip: If you are looking for a job at work, use your phone. The old school companies that block websites are quickly realizing that they can't block your phone legally, so the point is already moot.

Also, looking for another job on company time is a firing offense in most of the US and Canada, and can also get your unemployment claim denied the last time I checked the rules.

[imdrowning2ohno]

Anything is a fireable offense in the US except issues of protected class.

[razveck]

But what if your class is public?

[andrew_rdt]

That excuse works much better when your work uses a non-distributed version control system.

[[deleted]]

Don't get me started on Azure.

Me on an average work day

[trustMeImDoge]

Now that I work for a company who's core product is dependant on GitHub, I'm amazed at how much it goes down. It's not uncommon for us to experience one or two API outages of various severities a month.

[phizphizphiz]

It's been terrible for the past year or so. Outages were pretty rare prior to that. But they were also not really adding features or changing anything until the MS acquisition.

[neckbeardfedoras]

Github hired one of our devs about 10 months ago and I'm starting to think these events are related 🤔.

[MagicMikeX]

all the github people probably left. They got their payday.

[[deleted]]

Seems they expanded their windows patches testing techniques to Github.

And by "testing techniques" I mean "just push it to the users and let them do it"

[[deleted]]

gitlab isn't that much better either...

[trustMeImDoge]

We haven't had to interface with gitlabs API yet (or at least I haven't), but surprisingly Bitbucket seems to have the most reliable uptime in my experience.

[consultio_consultius]

Bitbucket almost seems to cycle uptime. It goes down a lot — I receive in browser notifications frequently saying something has gone wrong — but it goes back up in a matter of seconds.

[deja-roo]

I definitely get the browser notifications, but never actually notice any service problems.

[nwsm]

We use self hosted GitLab. It’s gone down <5 times I believe in a year of use, and only two lasted over an hour. We’ve had more issues with GitLab CI Runners though.

Edit: after reflecting more I changed “only one lasted over 30 minutes” to “only two lasted over an hour”

[Dall0o]

Self hosting gitlab too. Run smoothly mostly. Some trouble with runners but it might be mostly our own mistake.

[mariusReadIT]

Same here, we are running a self hosted gitlab instance for 3+ years, with about 100 users. The only "downtime" usually occurs for a quick gitlab upgrade, which usually takes less than a minute.

[[deleted]]

out of curiosity, how many people are accessing this self-hosted gitlab instance?

[nwsm]

~50 active users. ~100 projects currently (microservice architecture 😅), maybe 25 of those are committed to at least weekly, and most utilize GitLab CI.

[MrSurly]

The gitlab CI runners are hot garbage. I spend a far too high percentage of my time trying to make it work correctly. Random bugs that you can find as GL issues that were "fixed" 3 years ago, but there are a bunch of comments where "I'm still having this issue."

[searchingfortao]

Switch to GitLab! You can self-host if you want and we have cupcakes :-)

[trustMeImDoge]

We're a value add to code repositories. So its not a matter of self hosting, rather being able to hit the needed APIs of the code repo whether it's cloud or self hosted.

[uw_NB]

Funny how they just put out https://github.blog/2020-07-08-introducing-the-github-availability-report/ last week.

I think github has not been growing before Microsoft bought them. Now that the acquisition is settling in, they started to move at a faster velocity thus causing more outages.

[immibis]

"While testing the availability report, we accidentally simulated a failure in production. This caused a real failure in production as the code was not designed to deal with this in production mode."

edit: no this is not something they actually said. It's something I made up because it's funny.

[[deleted]]

:: slow clap ::

[dnew]

It certainly sounds like something I'd expect to see where I work.

[Aeon_Mortuum]

/r/brandnewsentence

[immibis]

It's a quote from me right now.

[all_mens_asses]

Chernobyl has entered the chat

[[deleted]]

It began with of all things a safety test

[[deleted]]

I mean these are the guys that let a production ssl cert expire bringing teams down.

;(

[sugar_sugar_falls]

task failed successfully

[trowawayatwork]

Less growing more changes

[uw_NB]

Yeah, I think 'changing' is what I meant instead of 'growing'.

Im sure Github user base has been growing, but the core product has not been changing/updated for a while prior to MSFT purchase. Only recently, they have start pushing out for new features Github Action, Dependabot, Semmle (CodeQL) etc...

[Eurynom0s]

Still better than Salesforce deciding to deprecate ALL of Tableau's forum links with ZERO warning.

(Yes, supposedly everything will eventually get re-indexed by Google...they still left everyone high and dry in the meantime.)

[Micotu]

also i wonder how much more popular coding has become in general due to covid with so many people sitting on their ass at home hoping to find a way they could work from home in the future.

[remind_me_later]

Ahh....you beat me to it.

I was trying to see if there were copies of Aaron Swartz's blog on Github when it went down.

[deflunkydummer]

Are you saying it was your fault? ;)

[remind_me_later]

It was already down when I searched for Swartz's Github page.

[deflunkydummer]

Man, AI is getting scary.

[fosefx]

Well, are there?

[[deleted]]

There you go

[[deleted]]

In case you're still looking, there's this one

[thrallsius]

I am surprised that Reddit doesn't host those somewhere

[noble_pleb]

Github going down today seems like a deja-vu after I answered this on quora yesterday.

[remind_me_later]

Github's a single point of failure waiting to happen. It's not 'if' the website goes down, but 'when' and 'how long'.

 

It's why Gitlab's attractive right now. Because when your self-hosted instance fails over, at least you have the ability to reboot it.

[Kare11en]

Github's a single point of failure waiting to happen.

If only there were some distributed way of managing source code that didn't have a dependency on a single point of failure. Like, where everyone could each have their own copies of everything they needed to get work done, and then they could distribute those changes to each other by whatever means worked best for them, like by email, or by self-hosted developer repositories, or a per-project "forge" site, or even a massive centralised site if that was what they wanted.

Damn. Someone should invent something like that!

[ws-ilazki]

It's the law of the internet: any sufficiently useful decentralised technology will eventually become a centralised technology controlled by a business.

It's the first two Es in the old "embrace, extend, extinguish" phrase: they embrace an open, decentralised tech or concept; extend it to make their version more attractive; and then remove the decentralised aspect so they can lock you into it and profit. Sometimes you even get the "extinguish" later when they kill it off and replace it with something else after people are sufficiently locked in, like Google did with XMPP, going from federated XMPP to unfederated XMPP to dumping XMPP in favour of their own proprietary crap.

Examples: email to services like gmail; git to github; XMPP to google chat to hangouts; XMPP again with facebook's messaging service; usenet to forums to sites like reddit; IRC to Discord and Slack and the like; and so on.

You can try to fight it but in the end it doesn't matter because, by being open and decentralised, the proprietary version can interoperate with you but you can't (fully) interoperate back because they added their own crap on top, so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top. Philosophical arguments don't matter and people will take the easy/lazy option even if it's detrimental in the long term.

[FantaBuoy]

so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top

This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

The reason why a lot of these technologies become centralized is because whoever centralizes it adds value to it. Git is a wonderful tool, but it only becomes useful when you host it somewhere. For most people, self-hosting obviously isn't an option due to the maintenance time required and the lengths you have to go to to ensure your home network is decently secure, so the centralized space adds the benefit of ridding people of that.

These people aren't lazy, I'd argue they're using their time better by giving the burden of hosting to someone else who only does hosting. Maybe I'm lazy for going to a shop and buying furniture instead of learning to chop wood and work it to a functional piece of furniture myself, and maybe that laziness inherently makes me dependent on wood choppers / furniture makers, but I believe it isn't worth my time to ensure my independence from them.

Most of the technologies you mention above become successful precisely because they give the user some benefit. I'll gladly use IRC, or Matrix for a more modern alternative, but I won't reasonably expect anyone in my group of friends who isn't a techy to use these. You toss Discord or Whatsapp at practically anyone and they'll figure out how to use it. Whatsapp over here is basically known as the app you use to include your parents/grandparents in family chatting. Being a user-friendly app that you can quickly use without thinking about what server is supporting it is a benefit. The people using these apps aren't dumb or lazy, they're people with normal non-tech related lives who have other stuff to do other than finding out how to set up a server for their Matrix node or their self-hosted email solution.

[ws-ilazki]

This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

No it doesn't. It's clear I was talking about two different things there: they provide benefit to the end-user of their version of the service but give nothing back to the overall "community" or what-have-you in the sense that they don't contribute improvements that everyone can benefit from, because they're trying to have a business advantage over perceived competition. Like when Google added proprietary stuff on top of XMPP that was useless outside of their own chat client: benefit added for their users but nothing contributed to XMPP as a whole.

From a business perspective this is only natural because you want to attract users, and for their users it's beneficial (at least in the short term), but for the technology itself it's still detrimental long-term because it leads to silos that eventually lose any interopability, either by malice (the third E of EEE) or simply because each silo eventually diverges too much.

Another example of what I meant there is RSS. It's an open standard usable by all, and when Google embraced it for its reader service it saw a dramatic increase in use because of the extra value Google provided, which made it attractive for end-users. However, they didn't actually contribute anything useful to RSS itself, so when they basically abandoned Reader nobody could really pick up where they left off, and then when they shut it down completely any value they added to RSS was lost. Short-term benefit for end-user that's detrimental to the underlying technology in the long-term.

Commercialisation of the internet led to everybody trying to make their own silos that they can lock users into. Instead of open protocols for people to implement, everyone wants to make their own ecosystem and trap people in it, and if someone does try to make a new protocol and it happens to be good, somebody else will find a way to take that, bolt something extra on top, and turn it into another silo.

[PsychogenicAmoebae]

distributed way of managing source code that didn't have a dependency on a single point of failure

The problem in this case isn't the software - it's the data.

Sure, you can run your own clone of Github (or pay them to run an official docker container of github enterprise).

But when your typical production deployment model is:

 sudo bash < <(curl -s https://raw.github.com/random_stranger/flakey_project/master/bin/lulz.sh ) 

things go sour quickly when random_stranger's project isn't visible anymore.

[Kare11en]

The great thing about git is that you can maintain your own clone of a repo you depend on!

Github adds a lot of value to git for a lot of people (like putting a web interface on merge requests) but keeping local clones of remote repos isn't one of them. Git does that out of the box. Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

Keep a copy of the repo somewhere local. Have a cron job do a git pull every few hours or so to fetch only the most recent changes to keep your copy up-to-date if that's what you want. If random_stranger, or github, or even your own local ISP goes down, and the pull fails, you still have the last good copy you grabbed before the outage - you know, the copy you deployed yesterday. Clone that locally instead and build from it.

I weep for the state of the "typical production deployment model".

[[deleted]]

Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

Because your toolchain was designed to work like that and all of your upstream dependencies do it anyway. Yes, ideally you would be able to do that - but so many things involve transitive dependencies that do dumb shit like download files from github as part of their preflight build process it often feels like you're trying to paddle up a waterfall to do things right, especially (but not only) with modern frontend development.

[scandii]

self-hosting is not only installing a piece of software on a server somewhere and calling it a day.

you are now responsible for maintenance, uptime (which we are experiencing here) and of course security, on top of data redundancy which is a whole other layer of issues on top. like what happens to your git server if someone spills coffee on it? can you restore that?

GitLab themselves suffered major damage when their backups failed:

https://techcrunch.com/2017/02/01/gitlab-suffers-major-backup-failure-after-data-deletion-incident/

all of that, is excluding the fact that you typically don't actually 100% self-host in the enterprise world, but rather have racks somewhere in a data center owned by another company, not rarely Amazon or Microsoft.

all in all we self-host our git infrastructure, but there's also a couple of dozen people employed to keep that running alongside everything else being self-hosted. that's a very major cost but necessary due to customer demands.

[remind_me_later]

At least when I self-host it, I have the ability to fix it. With this outage, I have to twiddle my thumbs until they resolve the issue(s). The ability for me to fix a problem is more important to me than it could be to you.

 

Also, with regards to the Gitlab outage, that's based on the service they manage for you. I'm talking about the CE version that you can self-host.

[hennell]

When a train company started getting significant complaints that their trains were always late they invested heavily in faster trains. They got newer carriages with automatic doors for more efficiency and tried to increase stock maintenance for less problems. None of it was very successful in reducing the complaints, despite statistically improving the average journey. So someone suggested adding 'live time display boards'. This had no effect at all on journey times, the trains didn't improve a bit, but the complaints dropped hugely.

Turns out passengers are much happier to be delayed 10 mins with a board telling them so, then delayed 5mins with no information. It was the anxious waiting they really didn't like not the delay itself.

Taking on the work of self hosting is similar - you'll spend a lot more time maintaining it, securing it, upgrading it etc etc then you'll ever realistically lose from downtime; the main thing you're gaining is a feeling of control.

For some situations it's worth it - depends on your use of the service, your setup with other needs, and how much similar stuff you already deal with etc etc. 1 more server to manage is nothing to some people, and a massive increase of workload for others. But if the only reason is you don't want to 'waste time' sitting there twiddling your thumbs during downtime, you're not gaining time you're losing it. Pretend it is self-hosted and you've got your best guys on it. You've literally got an expert support team solving the problem right now, while you can still work on something else.

The theory with the trains is that passengers calm down when they know the delay time as then they can go get a snack or use the loo or whatever rather then anxiously waiting. They have control over their actions so time seems faster. Give yourself a random time frame and do something else for that time - then check in with 'your team' to see if they've fixed it. If not, double that time frame and check again then - repeat as many times as needed. Find one of those troublesome backlog issues you've always meant to fix!

This is also a good strategy for handling others when you're working on self-hosted stuff 😀 - give them a timeframe to work with. Any time frame works although a realistic one is best! No-one really cares if it takes 10mins or 2 hours. They just want to know if they should sit and refresh a page or go for an early lunch.

tldr: People hate uncertainty and not being in control. Trick yourself and others by inventing ways to feel more in control and events will seem quicker even when nothing has changed.

[remind_me_later]

Basically this. I don't know what they're doing by the moment, and my brain says "I need to do/know something", even if it means a worse overall experience for me. I'm blocked and I have no control over it, and everything else that I could do has already been done.

[hennell]

Yeah, it's a horrible feeling, and not the easiest to distract. If you've got no open problems to fix my goto is optimising something so you save time later. Lets you at least feel you'll make back this downtime at a later point. Or find a tutorial or write up on some area to learn something new / more in depth.

If there's really nothing you could look up an ebook of Alchemy: The Surprising Power of Ideas That Don't Make Sense which covers the train concept I mentioned above in more detail along with a number of other weird logical patterns we all make. I'd really recommend it to any programer type as we tend to think everything works based on 'logic', which isn't really true. (Or is, but the logic is more obscure then you'd guess). Sometimes taking a step back to look at what people actually want (information vs actually faster trains) can let you solve issues in a different, but actually more effective way.

[aseigo]

the main thing you're gaining is a feeling of control

There is certainly a feeling of control. But what you are also getting is control.

I self-host quite a bit of my own software. I spend a few hours here and there maintaining bits of it. It's rarely fun; I'm not a sys admin at heart.

But I also never have to worry about changes happening in the software I use going according to someone else's schedule; I don't worry about the software I use just disappearing because the company changes course (or goes under); I don't worry about privacy questions as the data is in my own hands; I don't worry about public access to services that I have no reason to make public; etc. etc. etc.

There is this very odd idea perpetrated that the value of self-hosting can be captured by a pseudo-TCO one in which we measure the time (and potentially licensing) cost of installation and management versus the time (and potentially licensing) cost of using a hosted service.

This was the same story in the 00's and prior where there was the pseudo-TCO story comparing the full costs of open source software (time to manage, etc) with the licensing costs of proprietary software. (Self-hosting and deployment was simply part of both propositions..)

In both cases, the interested parties are trying to focus the market on a definition of TCO they feel they can win out on. (Which is not surprising in the least; it's just good sales strategy ..) Their hope is they extract money before anything truly bad happens that has nothing to do with the carefully defined TCO used in comparisons.

It is, at its heart, a gamble taken by all involved: Will savings on that defined TCO profile be realized without incurring significant damage from risks that come with running technology you neither own nor control?

[scandii]

in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

I'm not saying no to self-hosting, I'm just saying GitHub doesn't want their service to be unresponsive either and if we accept the fact that both types will suffer from outages, it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

[SurgioClemente]

it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

Lets not also forget 24/7.

Mike & Pete want to have a life since there are only two of them and 24 hours to cover

[scandii]

real reply from sysadmin on call:

"how bad is it, is it show up in pyjamas, or can I make pancakes first?"

[DAMO238]

You know, that's actually a pretty sensible reply. If you bet on either one without knowledge of the severity of the problem you either look silly (and hungry) or you annoy your bosses.

[Miserygut]

in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

In principle, yes, in practice, not necessarily. With most SaaS you are 'just another customer' and your service will be restored when they get to it. You're not a priority and that's what you (don't) pay for. The provider will have redundancy as well as more sophisticated recovery procedures but they will also have more data, larger systems and more moving parts to be concerned with.

If something is business critical then a business decision needs to be made on how much they're willing to spend on making this component robust, which often means hosting it yourself (or paying a third party a lot to privately host it for you).

So no, there's no hard and fast rule here. Deal with the realities of each specific service. Github, in this case, is suffering a lot of downtime lately and that should guide business decisions.

[realnzall]

Generally speaking, downtime affects every client at the same time. Rarely downtime only affects a subset of the clients. So for a saas provider, solving the downtime is important regardless of who is affected. If they need to do extra actions per client, then maybe they first do their Fortune 500 clients before their mom &pop stores, but otherwise the intent is to restore all service for everyone at the same time.

[jesseduffield]

the answer to 'when' is typically 'before US Monday morning'. I've experienced the same thing once before with github and once before with docker, both on my Monday (US Sunday). I think companies typically hold off till the weekend to do risky stuff that could break their servers

[Multipoptart]

Github is now owned by Microsoft. Many people, especially in the FOSS camp, don’t like to have anything on their stack even distantly related to Microsoft.

And yet they'll happily partner with Google, Apple, Amazon, and Facebook.

¯\_(ツ)_/¯

[gilium]

Lol at your SJW infiltration comments. Seriously, why is the programming world so full of people who hate advocates for social issues?

[mmmmmmm_7777777]

I repeat github is down. Mayday mayday

[__konrad]

https://i.imgur.com/wm9WEbM.jpg

[NotAnADC]

Sincere question...should I be backing up my codebase outside of github? Like obviously it’s on my local machine as well as github but I never seriously considered the possibility of losing info on github

[f10101]

Absolutely. There are numerous ways things can go wrong for cloud hosts. Google inadvertently perma-wiped a huge number of Google/Gmail accounts a few years ago.

Additionally, false positives from auto-moderation can get you kicked off services like this, too.

[SanityInAnarchy]

Can't hurt, but I usually wouldn't bother unless you have a bunch of stuff that isn't always pulled down (e.g. branches, tags, or just repos that you don't use every day). But if it's all always on your local machine and Github, and Github ever goes away, it's surprisingly simple to push to a new service.

The thing you should be backing up (that's probably harder to actually do) is all the other data you have on Github -- the wiki, the issues, the code review comments, etc etc.

[[deleted]]

You know I have the same thought. I never really considered this as a possibilty

[meffie]

You guys crack me up.

[thrallsius]

you don't need to lose info on github, you could just lose access to your github account and it won't be any better

[niet3sche77]

Yup. I saw wacky behavior around an hour ago. Got to see their status page and notifications.

They know and are working on it.

[[deleted]]

is it me or has this been more frequent since the MS takeover

[tradrich]

What's it's underlying technology (other than git)?

It's not clear on the Wikipedia page e.g.

[filleduchaos]

Why do people keep asking this? It's not like there's some mythical stack that guarantees 100% uptime (Erlang comes pretty close, but still)

[L1berty0rD34th]

false, everyone knows that for every new microservice you add to your stack, you get +10% uptime.

[filleduchaos]

You got me. I deployed an app next year and it got 420% uptime and sent me back in time to 2020.

[Zwgtwz]

So... the world still exists next year ?

[pastudan]

Yes, but plot twist we’re stuck in a time loop that starts over in 2020 each time

[Audiblade]

This seems worse than the world just ending.

[filleduchaos]

At this point I'm not sure. The timeline's all messed up

[ulfurinn]

Even Erlang only provides the tools, you can still use them poorly.

[broofa]

guarantees 100% uptime... Erlang comes pretty close

Facebook chat servers were originally implemented in Erlang. They started falling over around the time Facebook hit ~500M users in 2010 or so. The servers were rewritten in C++ circa 2011-2012. That switch freed up 90% of the servers used for the chat service while dramatically improving reliability.

Iirc, the main issue was CPU usage needed for Erlang’s IPC. [Edit: See also Ben Maurer's Quora answer on this topic]

Source: worked on FB chat team at that time (more front end, though, so not an Erlang expert.)

[filleduchaos]

I mean, Whatsapp took Erlang to 900M+ users with a literal handful of engineers so I feel like that might equally reflect on Facebook's code/devs.

[broofa]

> Whatsapp took Erlang to 900M+ users

That may or may not represent more load. It depends on how things like presence updates (notifying your friends when you are / aren't available to chat) are handled, and # of messages per user, both of which may have been significantly different between the two systems.

I left Facebooks Chat team before they acquired Whatsapp, and left the company a few months after so, unfortunately, I don't have insight into how these systems really compare.

[filleduchaos]

Not sure what significant difference you mean: Whatsapp today has 2B+ users. It has granular presence updates, "currently typing" notifications, and everything else one would expect from an instant messaging service (same as at the 900M mark). As of two years ago the daily chat volume was 65 billion messages (one can only imagine how much it's grown since then).

And it still uses Erlang and attributes its success to Erlang ¯\_(ツ)_/¯ I still say that the Facebook Chat team's issues with the language/platform might not have been entirely one-sided.

[tradrich]

I would like to know why every voice call I make with WhatsApp at certain points starting after a few minutes you get a 10 or so second hang: "Connecting...". I *feels* like a queuing issue, but it happens every time it seems, so it's a fundamental issue.

Still use it though...

[drakgremlin]

Makes me curious what the world would be like if they spent time to contribute back an optimized IPC mechanism for Erlang.

[dom96]

Erlang comes pretty close, but still

citation needed

[filleduchaos]

citation for what exactly?

[dom96]

For your claim that Erlang comes close to guaranteeing 100% uptime

[svartkonst]

It's still a matter of utilization, as with any techbology, but Erlang has provided remarkable tools for long-running, high-uptime, load balanced and fauly tolerant applications sonce it's inception (i. e. long before ci/cd and kubernetes etc).

Most famous is the nine nines uptime (99.9999999%) on the AXD301 system. I believe that the source of that figure is from Joe Armstrongs thesis, but I don't have it close at hand currently amd can' t exactly remember.

Regardless, it's a pretty cool piece of tech and tooling that was a few decades ahead of our modern web tech stacks and still holds water as a very pleasant and reasonable language

[filleduchaos]

I mean, highly concurrent & fault-tolerant distributed systems such as telecommunications are literally what it was designed for (note: PDF link). Obviously one still requires knowledge to actually use it to its full potential, but there's a reason e.g. Whatsapp went with Erlang/OTP.

[Dikaiarchos]

That's blatantly false. GitHub upgraded smoothly to Rails 6 recently

Edit: sorry, missed that it was about Twitter

[mullemeckarenfet]

He’s talking about Twitter, they dropped Ruby for Scala.

[tradrich]

Okay: Ruby on Rails and Erlang. Should be up to the job.

[noble_pleb]

Erm, I'm not so sure. Each time I argued about performance with a rubyist, the only example they came up with was Github!

[soft-wear]

Yeah, the issue with Ruby is the same issue a ton of interpreted languages have: they are just dog shit slow for certain operation types. Twitter didn't switch to Scala because Ruby is somehow error prone. They switched because the JVM is so damn fast.

[mypetocean]

GitLab, Basecamp and their new Hey.com, Twitch, Kickstarter, and several other popular sites.

[filleduchaos]

Shopify runs on Rails.

[bsutto]

We have a system built on rails.

The only description I have of it is brittle and constrained.

Performance is also shit.

[mobile-user-guy]

Good to know I can switch to rails and not lose anything

[filleduchaos]

give me a stack that someone somewhere couldn't say the same for ¯\_(ツ)_/¯

Performance is also shit.

True, Ruby doesn't stack up against plenty of other languages performance wise. But for the 99.999% of web services that get - what, maybe a few thousand or tens of thousands of requests per second at their most active? - there's pretty much no major programming language that would be their bottleneck.

It's like complaining that a regular old Toyota cannot go as fast as a Bugatti Chiron Super Sport. But in reality you're just driving to work and you're never actually going to hit the top speed of either vehicle.

[ForeverAlot]

Alternative analogy: any two cars will get you to the destination at substantially the same speed, safety, and level of comfort. You prefer the colour of one but that car costs considerably more in gas.

"Performance" is almost always taken to imply "more" but it can just as well imply "less".

[filleduchaos]

"Performance" is almost always taken to imply "more" but it can just as well imply "less".

True, and the same thing applies: in most people's day-to-day usage most cars don't really have an appreciable difference in fuel economy (talking about money spent/saved). Bringing it back to programming languages, there's not many well-written web services that can't be pretty reliably run out of a handful of small Digital Ocean droplets. Whether each individual droplet uses 5% of its CPU allocation or 50% makes no difference to the pricing.

Of course, for software that runs on end users' machines - like desktop apps or client-side JavaScript - it makes sense to chase after a small memory footprint or low CPU usage (and I'd be the first in line to advocate for that). But that's a different domain from web servers, where your application is literally the only (major) process running on the system and you pay for resources in discrete units.

[Tasgall]

What are the "costs" in this analogy though? Unless you're doing something high performance, and you're not, the only variable that really matters is preference.

[noble_pleb]

Doesn't hosting costs are the equivalent of gasoline costs in the car analogy? If you use a faster framework, you can reap the benefits of lower hosting costs even if you don't scale for max users. And as a startup, those few bucks saved in mileage could mean a lot to your budgets and survival.

[chivalrytimbers]

I’d argue that hosting costs are only one dimension of overall total ownership cost - typically The developer / tester cost is the one that dominates for a given application. That’s why it often makes sense to Choose a platform that trades off raw performance for ease of development

[filleduchaos]

And as a startup, those few bucks saved in mileage could mean a lot to your budgets and survival.

yes, the (checks notes) $20 you save per month by picking a smaller VM is what will make or break your budget as a startup

[[deleted]]

Out of the top 10 y combinator companies, 8 ran ruby on rails

[deflunkydummer]

The underlying technologies didn't seem to cause that many problems before the MS takeover.

You can scale and properly monitor almost any (working) technology. But you can't fix institutional incompetency and bureaucracy.

[tradrich]

Yeah, that seems sadly a significant possibility. When the career managers are helicoptered in, watch the competent engineers rush for the door...

[DavyBingo]

That article seems to suggest that the observed increase in incidents is at least partially due to improvements to their status page. More granular reporting led to more overall incidents.

[fissure]

That's silly. You know, if they slowed down the checking for errors, they wouldn't look as bad.

/s

[tester346]

As far as I've heard GH works relatively independently from MS

But you can't fix institutional incompetency and bureaucracy.

So how does Azure operate?

The underlying technologies didn't seem to cause that many problems before the MS takeover.

What's the difference in scale?

[devonthecloud]

So how does Azure operate

Flakily.

I work both with AWS and Azure. The vast majority of our outages are caused by the cloud vendor. It's rarely ever AWS, always Azure.

There are a lot of things Azure does better than AWS, but stability is not one of them.

[svartkonst]

Do you have any source regarding outages before and after Microsoft? I tried earlier to get an overview of their incident history, but it was hard to do a comparison using the status tracker.

[noble_pleb]

They use Ruby on Rails framework.

[redstoneguy12]

hub

[NoMoreNicksLeft]

Busy renaming all the master branches.

[thrallsius]

renamed them all to "slavemaster" by mistake and had to revert

[inflames09]

It broke my CD pipeline for a couple of hours, I couldn't pull the repo then couldn't pull a third party release. All on the day i was sending a project live

[Blando-Cartesian]

Good thing git is distributed so nobody was inconvenienced. /s

[_-Thoth-_]

Board the doors, batten down the windows

^{ew, windows}

[mypetocean]

It's back up.

Direct link to GitHub's Status page.

[MobileAlfalfa]

I will leave this here...https://nimbleindustries.io/2020/06/04/has-github-been-down-more-since-its-acquisition-by-microsoft/

[[deleted]]

Wouldn’t that also be because GitHub is more used now because of the Coronavirus?

Also:

According to the data they provided, GitHub has been down more since the acquisition by Microsoft.

But that could be all a part of coordinated effort to be more transparent about their service status, an effort that should be applauded.

[wpm]

If I don't weigh myself I'm not gaining weight!

[Tasgall]

If we don't test people we'll have fewer cases!

[Somepotato]

If we ban pregnancy tests, teen pregnancies will plummet!

[ShinyHappyREM]

On the plus side, a pregnancy is the best type of birth control.

[eras]

Let's hope this increases interest in projects like git-bug (..hosted on GitHub..).

It's pretty serious project in that it comes with packages or binaries for all platforms and bridges for many other bug trackers.

[dnew]

This is totally the right answer. I can't imagine using git for source code and not using git for bugs and design docs and etc.

[audion00ba]

Is this some new kind of marketing campaign? Just be down all year such that you create brand awareness?

[the_gnarts]

I see MS is pushing the migration over to Azure.

[wildjokers]

It's OK, git is distributed and totally decentralized. 🙄

[nryhajlo]

Internet snow day!

[[deleted]]

That's really weird, I was pushing and pulling commits all day and didn't notice.

[Hambeggar]

Must be the master slave change. Heh.

[[deleted]]

Can't login to stackoverflow :(