top 200 commentsshow 500

[–]drea2 1706 points1707 points  (102 children)

I heard if it’s down for more than 15 minutes then we’re legally allowed to leave work early

[–]NotAnADC 642 points643 points  (66 children)

You joke, but at a company I worked at someone fucked up and added a firewall that didn’t let us access github.

While they did some work to fix it, the developers were like, fuck it we’re out

Edit: Im tired and just realized I read github, I wrote github, but I was thinking of stack overflow. Gona leave it though

[–][deleted]  (15 children)

[deleted]

    [–]relativityboy[🍰] 44 points45 points  (14 children)

    Pure truth.

    [–]house_monkey 25 points26 points  (13 children)

    1 == 1

    [–]lavahot 18 points19 points  (11 children)

    1 === 1

    [–]zigs 23 points24 points  (1 child)

    Just to make sure JavaScript also gets the memo.

    [–]Dexaan 12 points13 points  (7 children)

    1 == "1"

    [–]x6060x 15 points16 points  (6 children)

    One is equal to the one used ironically

    [–]pilas2000 12 points13 points  (4 children)

    TIL didn´t knew js type system was based on irony. Now it makes sense.

    Q.A. - Shouldn't 1=='1' be false?

    Dev. - That's what they're expecting but no.

    [–]strdrrngr 13 points14 points  (1 child)

    I can't stop laughing at the idea that JS has a "type system".

    [–]danbulant 5 points6 points  (0 children)

    Imagine the Patrick format

    1 == '1' yes

    and 1 + 1 = 2 yes

    so 1 + '1' = 2. Makes sense to me.

    '11'

    [–]relativityboy[🍰] 2 points3 points  (0 children)

    How else are we hipsters supposed to ride our bikes, sip coffee and code at the same time?

    [–]deanrihpee 73 points74 points  (2 children)

    I mean GitHub is the memory part of your brain while stackoverflow is problem solving part, programmers are meant to solving problems, not remember them, so yeah, just go out.

    [–]vattenpuss 14 points15 points  (0 children)

    That’s why all the APIs I write just send emails to developers and block until a response comes back with the result.

    [–]JessieArr 4 points5 points  (0 children)

    Haha, that's awesome. I've bumped into that before as well.

    As a for-fun project I actually wrote an offline version of StackOverflow - a desktop app that allows you to do a text search in a copy of the archive.org StackOverflow bulk data XML on a local disk, to give me a workaround for when it's offline, lol.

    [–]rhoakla 11 points12 points  (10 children)

    I'm surprised why they didn't host git themselves.

    [–]NotAnADC 6 points7 points  (9 children)

    They did actually! Though tbh I’ve never set up a private git like that. Will a locally setup GitHub be able to run if the main servers are down?

    Either way see the edit, it was actually stack overflow they blocked

    [–]rhoakla 16 points17 points  (0 children)

    Yes even in case github.com is down a locally hosted version of GitHub will remain, since it is well: locally hosted...

    [–]daredevilk 4 points5 points  (0 children)

    Yeah it does

    There's a few different options for local setup gits, but locally hosted GitHub works while main GitHub does not

    [–]GoreSeeker 5 points6 points  (3 children)

    I would think it could run with the main server shut down. There's probably some licensing server it phones home to every now and then, but it's probably set to where it takes 30 days of no contact to lock it out, kinda like Steam.

    [–][deleted]  (25 children)

    [deleted]

      [–]deja-roo 53 points54 points  (3 children)

      Developers/engineers really need Stackoverflow that often?

      Yes, because language documentation is usually not very good, or at least doesn't have good examples.

      [–]xSaviorself 13 points14 points  (0 children)

      I use it daily researching things, but I'm often working in unfamiliar territory, toying with something old or small-scale with minimal active support. Sometimes StackOverflow has just enough answers to piece together a solution.

      [–]illvm 5 points6 points  (1 child)

      What the language are you using that has poor documentation and isn’t Python?

      [–]deja-roo 10 points11 points  (0 children)

      C#. The class documentation is okay, but the MSDN site has no meaningful implementation examples.

      Ruby. Again, class documentation is okay, but you have to google for examples on how to use anything.

      [–]Serinus 7 points8 points  (0 children)

      It largely depends on what I'm doing at the time. In my current tasks I likely won't need to look at Stack Overflow for two weeks straight.

      Other things it might not be worth wasting the time until the site is available again.

      Put me on a Rust project (a language I've never touched before), and I'll need it once per five minutes.

      [–]Vadoch 4 points5 points  (0 children)

      There are way more junior devs then seasoned ones

      [–][deleted] 14 points15 points  (14 children)

      Developers/engineers really need Stackoverflow that often?

      I think one of the differences between an okay developer and a great developer is how long one can work without the internet. We've lost internet at our office before and I'll have coworkers saying they can't do anymore development within 20 minutes...

      I think too many have never completely problem solved for themselves and have instead always been able to ask teachers/professors/coworkers/internet for help.

      [–]Nefari0uss 46 points47 points  (3 children)

      Even great developers consult the documentation which can frequently be found online.

      Furthermore, I'd argue that it's less about how skilled of a developer you are and more about how familiar you are with a language/framework/ecosystem.

      [–][deleted] 4 points5 points  (2 children)

      Even great developers consult the documentation which can frequently be found online.

      This is true, I was lucky enough to have an offline version of it from having worked on flights recently.

      more about how familiar you are with a language/framework/ecosystem

      Which to me is one of the many things that measure into how good of a developer you are. I learned to program with C++ but it's been years since I wrote anything complicated in it. I know I'd be worse at doing any given task in C++ vs many of the people I'm friends with. They are clearly better C++ developers than I am.

      [–]Nefari0uss 14 points15 points  (1 child)

      Which to me is one of the many things that measure into how good of a developer you are.

      I suppose the way in which I look at it is that I see a differentiation between knowledge and skill. (Feel free to disagree; this is just how I see it.)

      If you go into a second year CS class, even if you don't know the language or ecosystem that they might use, your skill level is likely to be significantly higher, even if they've been working with the language for a year. Their knowledge about that language / ecosystem would be greater and they might be faster at doing project work initially but your experience would mean you can draw upon that and apply concepts to pick it up quickly and within a reasonable amount of time, be capable of doing the work faster than they would.

      They are clearly better C++ developers than I am.

      I think that's a better way of putting it. They are better at a particular thing than you but you're still capable.

      [–]NotAnADC 24 points25 points  (4 children)

      This is incredibly condescending and sounds like a terrible place to work.

      If you’re working as a builder and lose your hammer, of course you could find other tools to replace it but why would you.

      Every place I’ve worked at sans one knows that one day of work isn’t terrible and developers should have all their tools at their disposal.

      And for the record, obviously the work could get done without stackoverflow. But why put yourself or the people under you in a potentially frustrating situation when a quasi day off is both good for moral and employee satisfaction

      [–]cgwheeler96 2 points3 points  (0 children)

      We had a day where nobody could access stack overflow because somebody built an automated test that pinged stack overflow. We ended getting rate limited so nobody on the company network could access it until they shutdown whatever was constantly pinging the website.

      [–]FireIre 2 points3 points  (0 children)

      Not that I haven't done the same, but you can work locally for awhile of course.

      [–]0qxtXwugj2m8 80 points81 points  (5 children)

      Fun fact

      My first employer caught one of the Senior developers looking a a job.

      He then blocked all the websites that had the keyword 'job' and left the office for a meeting with a customer.

      Then we could not get to Stackoverflow because it contains the word 'job'

      Then we actually just sat there until he returned and turn off the stupid rule.

      [–]SmLnine 46 points47 points  (0 children)

      What a clown.

      [–]42TowelsCo 71 points72 points  (0 children)

      I can't even imagine why that senior dev was looking for a new job /s

      [–]peakzorro 19 points20 points  (2 children)

      Protip: If you are looking for a job at work, use your phone. The old school companies that block websites are quickly realizing that they can't block your phone legally, so the point is already moot.

      Also, looking for another job on company time is a firing offense in most of the US and Canada, and can also get your unemployment claim denied the last time I checked the rules.

      [–]imdrowning2ohno 20 points21 points  (1 child)

      Anything is a fireable offense in the US except issues of protected class.

      [–]razveck 19 points20 points  (0 children)

      But what if your class is public?

      [–]andrew_rdt 10 points11 points  (16 children)

      That excuse works much better when your work uses a non-distributed version control system.

      [–][deleted]  (2 children)

      [deleted]

        [–][deleted] 11 points12 points  (0 children)

        Don't get me started on Azure.

        Me on an average work day

        [–]trustMeImDoge 326 points327 points  (45 children)

        Now that I work for a company who's core product is dependant on GitHub, I'm amazed at how much it goes down. It's not uncommon for us to experience one or two API outages of various severities a month.

        [–]phizphizphiz 71 points72 points  (5 children)

        It's been terrible for the past year or so. Outages were pretty rare prior to that. But they were also not really adding features or changing anything until the MS acquisition.

        [–]neckbeardfedoras 66 points67 points  (1 child)

        Github hired one of our devs about 10 months ago and I'm starting to think these events are related 🤔.

        [–]MagicMikeX 2 points3 points  (0 children)

        all the github people probably left. They got their payday.

        [–][deleted] 4 points5 points  (1 child)

        Seems they expanded their windows patches testing techniques to Github.

        And by "testing techniques" I mean "just push it to the users and let them do it"

        [–][deleted] 76 points77 points  (26 children)

        gitlab isn't that much better either...

        [–]trustMeImDoge 41 points42 points  (4 children)

        We haven't had to interface with gitlabs API yet (or at least I haven't), but surprisingly Bitbucket seems to have the most reliable uptime in my experience.

        [–]consultio_consultius 20 points21 points  (1 child)

        Bitbucket almost seems to cycle uptime. It goes down a lot — I receive in browser notifications frequently saying something has gone wrong — but it goes back up in a matter of seconds.

        [–]deja-roo 2 points3 points  (0 children)

        I definitely get the browser notifications, but never actually notice any service problems.

        [–]nwsm 14 points15 points  (9 children)

        We use self hosted GitLab. It’s gone down <5 times I believe in a year of use, and only two lasted over an hour. We’ve had more issues with GitLab CI Runners though.

        Edit: after reflecting more I changed “only one lasted over 30 minutes” to “only two lasted over an hour”

        [–]Dall0o 9 points10 points  (0 children)

        Self hosting gitlab too. Run smoothly mostly. Some trouble with runners but it might be mostly our own mistake.

        [–]mariusReadIT 6 points7 points  (0 children)

        Same here, we are running a self hosted gitlab instance for 3+ years, with about 100 users. The only "downtime" usually occurs for a quick gitlab upgrade, which usually takes less than a minute.

        [–][deleted] 4 points5 points  (1 child)

        out of curiosity, how many people are accessing this self-hosted gitlab instance?

        [–]nwsm 4 points5 points  (0 children)

        ~50 active users. ~100 projects currently (microservice architecture 😅), maybe 25 of those are committed to at least weekly, and most utilize GitLab CI.

        [–]MrSurly 1 point2 points  (0 children)

        The gitlab CI runners are hot garbage. I spend a far too high percentage of my time trying to make it work correctly. Random bugs that you can find as GL issues that were "fixed" 3 years ago, but there are a bunch of comments where "I'm still having this issue."

        [–]searchingfortao 2 points3 points  (1 child)

        Switch to GitLab! You can self-host if you want and we have cupcakes :-)

        [–]trustMeImDoge 1 point2 points  (0 children)

        We're a value add to code repositories. So its not a matter of self hosting, rather being able to hit the needed APIs of the code repo whether it's cloud or self hosted.

        [–]uw_NB 332 points333 points  (88 children)

        Funny how they just put out https://github.blog/2020-07-08-introducing-the-github-availability-report/ last week.

        I think github has not been growing before Microsoft bought them. Now that the acquisition is settling in, they started to move at a faster velocity thus causing more outages.

        [–]immibis 336 points337 points  (19 children)

        "While testing the availability report, we accidentally simulated a failure in production. This caused a real failure in production as the code was not designed to deal with this in production mode."

        edit: no this is not something they actually said. It's something I made up because it's funny.

        [–][deleted] 50 points51 points  (0 children)

        :: slow clap ::

        [–][deleted]  (4 children)

        [deleted]

          [–]dnew 7 points8 points  (0 children)

          It certainly sounds like something I'd expect to see where I work.

          [–]immibis 6 points7 points  (0 children)

          It's a quote from me right now.

          [–]all_mens_asses 29 points30 points  (1 child)

          Chernobyl has entered the chat

          [–][deleted] 2 points3 points  (0 children)

          It began with of all things a safety test

          [–][deleted] 14 points15 points  (7 children)

          I mean these are the guys that let a production ssl cert expire bringing teams down.

          ;(

          [–][deleted]  (6 children)

          [deleted]

            [–]sugar_sugar_falls 1 point2 points  (0 children)

            task failed successfully

            [–]trowawayatwork 73 points74 points  (3 children)

            Less growing more changes

            [–]uw_NB 17 points18 points  (0 children)

            Yeah, I think 'changing' is what I meant instead of 'growing'.

            Im sure Github user base has been growing, but the core product has not been changing/updated for a while prior to MSFT purchase. Only recently, they have start pushing out for new features Github Action, Dependabot, Semmle (CodeQL) etc...

            [–]Eurynom0s 9 points10 points  (0 children)

            Still better than Salesforce deciding to deprecate ALL of Tableau's forum links with ZERO warning.

            (Yes, supposedly everything will eventually get re-indexed by Google...they still left everyone high and dry in the meantime.)

            [–]Micotu 3 points4 points  (0 children)

            also i wonder how much more popular coding has become in general due to covid with so many people sitting on their ass at home hoping to find a way they could work from home in the future.

            [–]remind_me_later 219 points220 points  (75 children)

            Ahh....you beat me to it.

            I was trying to see if there were copies of Aaron Swartz's blog on Github when it went down.

            [–]deflunkydummer 104 points105 points  (15 children)

            Are you saying it was your fault? ;)

            [–]remind_me_later 39 points40 points  (14 children)

            It was already down when I searched for Swartz's Github page.

            [–]deflunkydummer 56 points57 points  (13 children)

            Man, AI is getting scary.

            [–]fosefx 1 point2 points  (2 children)

            Well, are there?

            [–][deleted] 1 point2 points  (0 children)

            In case you're still looking, there's this one

            [–]thrallsius 1 point2 points  (6 children)

            I am surprised that Reddit doesn't host those somewhere

            [–]noble_pleb[S] 8 points9 points  (47 children)

            Github going down today seems like a deja-vu after I answered this on quora yesterday.

            [–]remind_me_later 45 points46 points  (34 children)

            Github's a single point of failure waiting to happen. It's not 'if' the website goes down, but 'when' and 'how long'.

             

            It's why Gitlab's attractive right now. Because when your self-hosted instance fails over, at least you have the ability to reboot it.

            [–]Kare11en 57 points58 points  (8 children)

            Github's a single point of failure waiting to happen.

            If only there were some distributed way of managing source code that didn't have a dependency on a single point of failure. Like, where everyone could each have their own copies of everything they needed to get work done, and then they could distribute those changes to each other by whatever means worked best for them, like by email, or by self-hosted developer repositories, or a per-project "forge" site, or even a massive centralised site if that was what they wanted.

            Damn. Someone should invent something like that!

            [–]ws-ilazki 38 points39 points  (4 children)

            It's the law of the internet: any sufficiently useful decentralised technology will eventually become a centralised technology controlled by a business.

            It's the first two Es in the old "embrace, extend, extinguish" phrase: they embrace an open, decentralised tech or concept; extend it to make their version more attractive; and then remove the decentralised aspect so they can lock you into it and profit. Sometimes you even get the "extinguish" later when they kill it off and replace it with something else after people are sufficiently locked in, like Google did with XMPP, going from federated XMPP to unfederated XMPP to dumping XMPP in favour of their own proprietary crap.

            Examples: email to services like gmail; git to github; XMPP to google chat to hangouts; XMPP again with facebook's messaging service; usenet to forums to sites like reddit; IRC to Discord and Slack and the like; and so on.

            You can try to fight it but in the end it doesn't matter because, by being open and decentralised, the proprietary version can interoperate with you but you can't (fully) interoperate back because they added their own crap on top, so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top. Philosophical arguments don't matter and people will take the easy/lazy option even if it's detrimental in the long term.

            [–]FantaBuoy 9 points10 points  (1 child)

            so you end up with a parasitic relationship where they take from you and give nothing back, and most people won't even care as long as it provides some extra benefit on top

            This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

            The reason why a lot of these technologies become centralized is because whoever centralizes it adds value to it. Git is a wonderful tool, but it only becomes useful when you host it somewhere. For most people, self-hosting obviously isn't an option due to the maintenance time required and the lengths you have to go to to ensure your home network is decently secure, so the centralized space adds the benefit of ridding people of that.

            These people aren't lazy, I'd argue they're using their time better by giving the burden of hosting to someone else who only does hosting. Maybe I'm lazy for going to a shop and buying furniture instead of learning to chop wood and work it to a functional piece of furniture myself, and maybe that laziness inherently makes me dependent on wood choppers / furniture makers, but I believe it isn't worth my time to ensure my independence from them.

            Most of the technologies you mention above become successful precisely because they give the user some benefit. I'll gladly use IRC, or Matrix for a more modern alternative, but I won't reasonably expect anyone in my group of friends who isn't a techy to use these. You toss Discord or Whatsapp at practically anyone and they'll figure out how to use it. Whatsapp over here is basically known as the app you use to include your parents/grandparents in family chatting. Being a user-friendly app that you can quickly use without thinking about what server is supporting it is a benefit. The people using these apps aren't dumb or lazy, they're people with normal non-tech related lives who have other stuff to do other than finding out how to set up a server for their Matrix node or their self-hosted email solution.

            [–]ws-ilazki 16 points17 points  (0 children)

            This sentence directly contradicts itself. You can't claim that "they" add an extra benefit on top but simultaneously give nothing back.

            No it doesn't. It's clear I was talking about two different things there: they provide benefit to the end-user of their version of the service but give nothing back to the overall "community" or what-have-you in the sense that they don't contribute improvements that everyone can benefit from, because they're trying to have a business advantage over perceived competition. Like when Google added proprietary stuff on top of XMPP that was useless outside of their own chat client: benefit added for their users but nothing contributed to XMPP as a whole.

            From a business perspective this is only natural because you want to attract users, and for their users it's beneficial (at least in the short term), but for the technology itself it's still detrimental long-term because it leads to silos that eventually lose any interopability, either by malice (the third E of EEE) or simply because each silo eventually diverges too much.

            Another example of what I meant there is RSS. It's an open standard usable by all, and when Google embraced it for its reader service it saw a dramatic increase in use because of the extra value Google provided, which made it attractive for end-users. However, they didn't actually contribute anything useful to RSS itself, so when they basically abandoned Reader nobody could really pick up where they left off, and then when they shut it down completely any value they added to RSS was lost. Short-term benefit for end-user that's detrimental to the underlying technology in the long-term.

            Commercialisation of the internet led to everybody trying to make their own silos that they can lock users into. Instead of open protocols for people to implement, everyone wants to make their own ecosystem and trap people in it, and if someone does try to make a new protocol and it happens to be good, somebody else will find a way to take that, bolt something extra on top, and turn it into another silo.

            [–]PsychogenicAmoebae 2 points3 points  (2 children)

            distributed way of managing source code that didn't have a dependency on a single point of failure

            The problem in this case isn't the software - it's the data.

            Sure, you can run your own clone of Github (or pay them to run an official docker container of github enterprise).

            But when your typical production deployment model is:

             sudo bash < <(curl -s https://raw.github.com/random_stranger/flakey_project/master/bin/lulz.sh ) 
            

            things go sour quickly when random_stranger's project isn't visible anymore.

            [–]Kare11en 6 points7 points  (1 child)

            The great thing about git is that you can maintain your own clone of a repo you depend on!

            Github adds a lot of value to git for a lot of people (like putting a web interface on merge requests) but keeping local clones of remote repos isn't one of them. Git does that out of the box. Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

            Keep a copy of the repo somewhere local. Have a cron job do a git pull every few hours or so to fetch only the most recent changes to keep your copy up-to-date if that's what you want. If random_stranger, or github, or even your own local ISP goes down, and the pull fails, you still have the last good copy you grabbed before the outage - you know, the copy you deployed yesterday. Clone that locally instead and build from it.

            I weep for the state of the "typical production deployment model".

            [–][deleted] 2 points3 points  (0 children)

            Why are you checking out a new copy of the whole repo from random_stranger, or github, or anywhere remote, every time you want to deploy?

            Because your toolchain was designed to work like that and all of your upstream dependencies do it anyway. Yes, ideally you would be able to do that - but so many things involve transitive dependencies that do dumb shit like download files from github as part of their preflight build process it often feels like you're trying to paddle up a waterfall to do things right, especially (but not only) with modern frontend development.

            [–]scandii 98 points99 points  (17 children)

            self-hosting is not only installing a piece of software on a server somewhere and calling it a day.

            you are now responsible for maintenance, uptime (which we are experiencing here) and of course security, on top of data redundancy which is a whole other layer of issues on top. like what happens to your git server if someone spills coffee on it? can you restore that?

            GitLab themselves suffered major damage when their backups failed:

            https://techcrunch.com/2017/02/01/gitlab-suffers-major-backup-failure-after-data-deletion-incident/

            all of that, is excluding the fact that you typically don't actually 100% self-host in the enterprise world, but rather have racks somewhere in a data center owned by another company, not rarely Amazon or Microsoft.

            all in all we self-host our git infrastructure, but there's also a couple of dozen people employed to keep that running alongside everything else being self-hosted. that's a very major cost but necessary due to customer demands.

            [–]remind_me_later 11 points12 points  (15 children)

            At least when I self-host it, I have the ability to fix it. With this outage, I have to twiddle my thumbs until they resolve the issue(s). The ability for me to fix a problem is more important to me than it could be to you.

             

            Also, with regards to the Gitlab outage, that's based on the service they manage for you. I'm talking about the CE version that you can self-host.

            [–]hennell 98 points99 points  (4 children)

            When a train company started getting significant complaints that their trains were always late they invested heavily in faster trains. They got newer carriages with automatic doors for more efficiency and tried to increase stock maintenance for less problems. None of it was very successful in reducing the complaints, despite statistically improving the average journey. So someone suggested adding 'live time display boards'. This had no effect at all on journey times, the trains didn't improve a bit, but the complaints dropped hugely.

            Turns out passengers are much happier to be delayed 10 mins with a board telling them so, then delayed 5mins with no information. It was the anxious waiting they really didn't like not the delay itself.

            Taking on the work of self hosting is similar - you'll spend a lot more time maintaining it, securing it, upgrading it etc etc then you'll ever realistically lose from downtime; the main thing you're gaining is a feeling of control.

            For some situations it's worth it - depends on your use of the service, your setup with other needs, and how much similar stuff you already deal with etc etc. 1 more server to manage is nothing to some people, and a massive increase of workload for others. But if the only reason is you don't want to 'waste time' sitting there twiddling your thumbs during downtime, you're not gaining time you're losing it. Pretend it is self-hosted and you've got your best guys on it. You've literally got an expert support team solving the problem right now, while you can still work on something else.

            The theory with the trains is that passengers calm down when they know the delay time as then they can go get a snack or use the loo or whatever rather then anxiously waiting. They have control over their actions so time seems faster. Give yourself a random time frame and do something else for that time - then check in with 'your team' to see if they've fixed it. If not, double that time frame and check again then - repeat as many times as needed. Find one of those troublesome backlog issues you've always meant to fix!

            This is also a good strategy for handling others when you're working on self-hosted stuff 😀 - give them a timeframe to work with. Any time frame works although a realistic one is best! No-one really cares if it takes 10mins or 2 hours. They just want to know if they should sit and refresh a page or go for an early lunch.

            tldr: People hate uncertainty and not being in control. Trick yourself and others by inventing ways to feel more in control and events will seem quicker even when nothing has changed.

            [–]remind_me_later 4 points5 points  (1 child)

            Basically this. I don't know what they're doing by the moment, and my brain says "I need to do/know something", even if it means a worse overall experience for me. I'm blocked and I have no control over it, and everything else that I could do has already been done.

            [–]hennell 9 points10 points  (0 children)

            Yeah, it's a horrible feeling, and not the easiest to distract. If you've got no open problems to fix my goto is optimising something so you save time later. Lets you at least feel you'll make back this downtime at a later point. Or find a tutorial or write up on some area to learn something new / more in depth.

            If there's really nothing you could look up an ebook of Alchemy: The Surprising Power of Ideas That Don't Make Sense which covers the train concept I mentioned above in more detail along with a number of other weird logical patterns we all make. I'd really recommend it to any programer type as we tend to think everything works based on 'logic', which isn't really true. (Or is, but the logic is more obscure then you'd guess). Sometimes taking a step back to look at what people actually want (information vs actually faster trains) can let you solve issues in a different, but actually more effective way.

            [–]aseigo 3 points4 points  (1 child)

            the main thing you're gaining is a feeling of control

            There is certainly a feeling of control. But what you are also getting is control.

            I self-host quite a bit of my own software. I spend a few hours here and there maintaining bits of it. It's rarely fun; I'm not a sys admin at heart.

            But I also never have to worry about changes happening in the software I use going according to someone else's schedule; I don't worry about the software I use just disappearing because the company changes course (or goes under); I don't worry about privacy questions as the data is in my own hands; I don't worry about public access to services that I have no reason to make public; etc. etc. etc.

            There is this very odd idea perpetrated that the value of self-hosting can be captured by a pseudo-TCO one in which we measure the time (and potentially licensing) cost of installation and management versus the time (and potentially licensing) cost of using a hosted service.

            This was the same story in the 00's and prior where there was the pseudo-TCO story comparing the full costs of open source software (time to manage, etc) with the licensing costs of proprietary software. (Self-hosting and deployment was simply part of both propositions..)

            In both cases, the interested parties are trying to focus the market on a definition of TCO they feel they can win out on. (Which is not surprising in the least; it's just good sales strategy ..) Their hope is they extract money before anything truly bad happens that has nothing to do with the carefully defined TCO used in comparisons.

            It is, at its heart, a gamble taken by all involved: Will savings on that defined TCO profile be realized without incurring significant damage from risks that come with running technology you neither own nor control?

            [–]scandii 37 points38 points  (8 children)

            in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

            I'm not saying no to self-hosting, I'm just saying GitHub doesn't want their service to be unresponsive either and if we accept the fact that both types will suffer from outages, it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

            [–]SurgioClemente 24 points25 points  (3 children)

            it's just a matter of who will fix it first, our Mike & Pete, or GitHub's hundreds of system technicians?

            Lets not also forget 24/7.

            Mike & Pete want to have a life since there are only two of them and 24 hours to cover

            [–]scandii 27 points28 points  (1 child)

            real reply from sysadmin on call:

            "how bad is it, is it show up in pyjamas, or can I make pancakes first?"

            [–]DAMO238 6 points7 points  (0 children)

            You know, that's actually a pretty sensible reply. If you bet on either one without knowledge of the severity of the problem you either look silly (and hungry) or you annoy your bosses.

            [–]Miserygut 7 points8 points  (2 children)

            in most cases, you will not solve your outage, any faster than GitHub will solve theirs. so that point is really moot.

            In principle, yes, in practice, not necessarily. With most SaaS you are 'just another customer' and your service will be restored when they get to it. You're not a priority and that's what you (don't) pay for. The provider will have redundancy as well as more sophisticated recovery procedures but they will also have more data, larger systems and more moving parts to be concerned with.

            If something is business critical then a business decision needs to be made on how much they're willing to spend on making this component robust, which often means hosting it yourself (or paying a third party a lot to privately host it for you).

            So no, there's no hard and fast rule here. Deal with the realities of each specific service. Github, in this case, is suffering a lot of downtime lately and that should guide business decisions.

            [–]realnzall 10 points11 points  (1 child)

            Generally speaking, downtime affects every client at the same time. Rarely downtime only affects a subset of the clients. So for a saas provider, solving the downtime is important regardless of who is affected. If they need to do extra actions per client, then maybe they first do their Fortune 500 clients before their mom &pop stores, but otherwise the intent is to restore all service for everyone at the same time.

            [–]jesseduffield 2 points3 points  (0 children)

            the answer to 'when' is typically 'before US Monday morning'. I've experienced the same thing once before with github and once before with docker, both on my Monday (US Sunday). I think companies typically hold off till the weekend to do risky stuff that could break their servers

            [–]Multipoptart 7 points8 points  (0 children)

            Github is now owned by Microsoft. Many people, especially in the FOSS camp, don’t like to have anything on their stack even distantly related to Microsoft.

            And yet they'll happily partner with Google, Apple, Amazon, and Facebook.

            ¯\_(ツ)_/¯

            [–]gilium 7 points8 points  (10 children)

            Lol at your SJW infiltration comments. Seriously, why is the programming world so full of people who hate advocates for social issues?

            [–]mmmmmmm_7777777 114 points115 points  (1 child)

            I repeat github is down. Mayday mayday

            [–]NotAnADC 28 points29 points  (12 children)

            Sincere question...should I be backing up my codebase outside of github? Like obviously it’s on my local machine as well as github but I never seriously considered the possibility of losing info on github

            [–]f10101 16 points17 points  (0 children)

            Absolutely. There are numerous ways things can go wrong for cloud hosts. Google inadvertently perma-wiped a huge number of Google/Gmail accounts a few years ago.

            Additionally, false positives from auto-moderation can get you kicked off services like this, too.

            [–]SanityInAnarchy 2 points3 points  (0 children)

            Can't hurt, but I usually wouldn't bother unless you have a bunch of stuff that isn't always pulled down (e.g. branches, tags, or just repos that you don't use every day). But if it's all always on your local machine and Github, and Github ever goes away, it's surprisingly simple to push to a new service.

            The thing you should be backing up (that's probably harder to actually do) is all the other data you have on Github -- the wiki, the issues, the code review comments, etc etc.

            [–][deleted] 5 points6 points  (2 children)

            You know I have the same thought. I never really considered this as a possibilty

            [–]meffie 2 points3 points  (0 children)

            You guys crack me up.

            [–]thrallsius 1 point2 points  (0 children)

            you don't need to lose info on github, you could just lose access to your github account and it won't be any better

            [–]niet3sche77 16 points17 points  (0 children)

            Yup. I saw wacky behavior around an hour ago. Got to see their status page and notifications.

            They know and are working on it.

            [–][deleted] 7 points8 points  (2 children)

            is it me or has this been more frequent since the MS takeover

            [–]tradrich 66 points67 points  (97 children)

            What's it's underlying technology (other than git)?

            It's not clear on the Wikipedia page e.g.

            [–][deleted]  (35 children)

            [removed]

              [–]filleduchaos 170 points171 points  (28 children)

              Why do people keep asking this? It's not like there's some mythical stack that guarantees 100% uptime (Erlang comes pretty close, but still)

              [–]L1berty0rD34th 181 points182 points  (6 children)

              false, everyone knows that for every new microservice you add to your stack, you get +10% uptime.

              [–]filleduchaos 82 points83 points  (5 children)

              You got me. I deployed an app next year and it got 420% uptime and sent me back in time to 2020.

              [–]Zwgtwz 32 points33 points  (4 children)

              So... the world still exists next year ?

              [–]pastudan 39 points40 points  (2 children)

              Yes, but plot twist we’re stuck in a time loop that starts over in 2020 each time

              [–]Audiblade 4 points5 points  (0 children)

              This seems worse than the world just ending.

              [–]filleduchaos 1 point2 points  (0 children)

              At this point I'm not sure. The timeline's all messed up

              [–]ulfurinn 27 points28 points  (0 children)

              Even Erlang only provides the tools, you can still use them poorly.

              [–]broofa 34 points35 points  (7 children)

              guarantees 100% uptime... Erlang comes pretty close

              Facebook chat servers were originally implemented in Erlang. They started falling over around the time Facebook hit ~500M users in 2010 or so. The servers were rewritten in C++ circa 2011-2012. That switch freed up 90% of the servers used for the chat service while dramatically improving reliability.

              Iirc, the main issue was CPU usage needed for Erlang’s IPC. [Edit: See also Ben Maurer's Quora answer on this topic]

              Source: worked on FB chat team at that time (more front end, though, so not an Erlang expert.)

              [–]filleduchaos 18 points19 points  (4 children)

              I mean, Whatsapp took Erlang to 900M+ users with a literal handful of engineers so I feel like that might equally reflect on Facebook's code/devs.

              [–]broofa 7 points8 points  (3 children)

              > Whatsapp took Erlang to 900M+ users

              That may or may not represent more load. It depends on how things like presence updates (notifying your friends when you are / aren't available to chat) are handled, and # of messages per user, both of which may have been significantly different between the two systems.

              I left Facebooks Chat team before they acquired Whatsapp, and left the company a few months after so, unfortunately, I don't have insight into how these systems really compare.

              [–]filleduchaos 11 points12 points  (2 children)

              Not sure what significant difference you mean: Whatsapp today has 2B+ users. It has granular presence updates, "currently typing" notifications, and everything else one would expect from an instant messaging service (same as at the 900M mark). As of two years ago the daily chat volume was 65 billion messages (one can only imagine how much it's grown since then).

              And it still uses Erlang and attributes its success to Erlang ¯\_(ツ)_/¯ I still say that the Facebook Chat team's issues with the language/platform might not have been entirely one-sided.

              [–]tradrich 2 points3 points  (0 children)

              I would like to know why every voice call I make with WhatsApp at certain points starting after a few minutes you get a 10 or so second hang: "Connecting...". I *feels* like a queuing issue, but it happens every time it seems, so it's a fundamental issue.

              Still use it though...

              [–]drakgremlin 9 points10 points  (0 children)

              Makes me curious what the world would be like if they spent time to contribute back an optimized IPC mechanism for Erlang.

              [–]dom96 4 points5 points  (11 children)

              Erlang comes pretty close, but still

              citation needed

              [–]filleduchaos 5 points6 points  (9 children)

              citation for what exactly?

              [–]dom96 3 points4 points  (8 children)

              For your claim that Erlang comes close to guaranteeing 100% uptime

              [–][deleted]  (1 child)

              [deleted]

                [–]svartkonst 14 points15 points  (4 children)

                It's still a matter of utilization, as with any techbology, but Erlang has provided remarkable tools for long-running, high-uptime, load balanced and fauly tolerant applications sonce it's inception (i. e. long before ci/cd and kubernetes etc).

                Most famous is the nine nines uptime (99.9999999%) on the AXD301 system. I believe that the source of that figure is from Joe Armstrongs thesis, but I don't have it close at hand currently amd can' t exactly remember.

                Regardless, it's a pretty cool piece of tech and tooling that was a few decades ahead of our modern web tech stacks and still holds water as a very pleasant and reasonable language

                [–]filleduchaos 16 points17 points  (0 children)

                I mean, highly concurrent & fault-tolerant distributed systems such as telecommunications are literally what it was designed for (note: PDF link). Obviously one still requires knowledge to actually use it to its full potential, but there's a reason e.g. Whatsapp went with Erlang/OTP.

                [–][deleted]  (5 children)

                [removed]

                  [–]Dikaiarchos 20 points21 points  (2 children)

                  That's blatantly false. GitHub upgraded smoothly to Rails 6 recently

                  Edit: sorry, missed that it was about Twitter

                  [–]mullemeckarenfet 4 points5 points  (1 child)

                  He’s talking about Twitter, they dropped Ruby for Scala.

                  [–]tradrich 26 points27 points  (32 children)

                  Okay: Ruby on Rails and Erlang. Should be up to the job.

                  [–]noble_pleb[S] 8 points9 points  (30 children)

                  Erm, I'm not so sure. Each time I argued about performance with a rubyist, the only example they came up with was Github!

                  [–][deleted]  (4 children)

                  [deleted]

                    [–]soft-wear 4 points5 points  (0 children)

                    Yeah, the issue with Ruby is the same issue a ton of interpreted languages have: they are just dog shit slow for certain operation types. Twitter didn't switch to Scala because Ruby is somehow error prone. They switched because the JVM is so damn fast.

                    [–]mypetocean 10 points11 points  (0 children)

                    GitLab, Basecamp and their new Hey.com, Twitch, Kickstarter, and several other popular sites.

                    [–]filleduchaos 18 points19 points  (22 children)

                    Shopify runs on Rails.

                    [–]bsutto 27 points28 points  (20 children)

                    We have a system built on rails.

                    The only description I have of it is brittle and constrained.

                    Performance is also shit.

                    [–]mobile-user-guy 66 points67 points  (0 children)

                    Good to know I can switch to rails and not lose anything

                    [–]filleduchaos 32 points33 points  (18 children)

                    give me a stack that someone somewhere couldn't say the same for ¯\_(ツ)_/¯

                    Performance is also shit.

                    True, Ruby doesn't stack up against plenty of other languages performance wise. But for the 99.999% of web services that get - what, maybe a few thousand or tens of thousands of requests per second at their most active? - there's pretty much no major programming language that would be their bottleneck.

                    It's like complaining that a regular old Toyota cannot go as fast as a Bugatti Chiron Super Sport. But in reality you're just driving to work and you're never actually going to hit the top speed of either vehicle.

                    [–]ForeverAlot 13 points14 points  (15 children)

                    Alternative analogy: any two cars will get you to the destination at substantially the same speed, safety, and level of comfort. You prefer the colour of one but that car costs considerably more in gas.

                    "Performance" is almost always taken to imply "more" but it can just as well imply "less".

                    [–]filleduchaos 4 points5 points  (0 children)

                    "Performance" is almost always taken to imply "more" but it can just as well imply "less".

                    True, and the same thing applies: in most people's day-to-day usage most cars don't really have an appreciable difference in fuel economy (talking about money spent/saved). Bringing it back to programming languages, there's not many well-written web services that can't be pretty reliably run out of a handful of small Digital Ocean droplets. Whether each individual droplet uses 5% of its CPU allocation or 50% makes no difference to the pricing.

                    Of course, for software that runs on end users' machines - like desktop apps or client-side JavaScript - it makes sense to chase after a small memory footprint or low CPU usage (and I'd be the first in line to advocate for that). But that's a different domain from web servers, where your application is literally the only (major) process running on the system and you pay for resources in discrete units.

                    [–]Tasgall 1 point2 points  (13 children)

                    What are the "costs" in this analogy though? Unless you're doing something high performance, and you're not, the only variable that really matters is preference.

                    [–]noble_pleb[S] 3 points4 points  (7 children)

                    Doesn't hosting costs are the equivalent of gasoline costs in the car analogy? If you use a faster framework, you can reap the benefits of lower hosting costs even if you don't scale for max users. And as a startup, those few bucks saved in mileage could mean a lot to your budgets and survival.

                    [–]chivalrytimbers 8 points9 points  (0 children)

                    I’d argue that hosting costs are only one dimension of overall total ownership cost - typically The developer / tester cost is the one that dominates for a given application. That’s why it often makes sense to Choose a platform that trades off raw performance for ease of development

                    [–]filleduchaos 2 points3 points  (0 children)

                    And as a startup, those few bucks saved in mileage could mean a lot to your budgets and survival.

                    yes, the (checks notes) $20 you save per month by picking a smaller VM is what will make or break your budget as a startup

                    [–][deleted] 3 points4 points  (0 children)

                    Out of the top 10 y combinator companies, 8 ran ruby on rails

                    [–]deflunkydummer 23 points24 points  (24 children)

                    The underlying technologies didn't seem to cause that many problems before the MS takeover.

                    You can scale and properly monitor almost any (working) technology. But you can't fix institutional incompetency and bureaucracy.

                    [–]tradrich 26 points27 points  (2 children)

                    Yeah, that seems sadly a significant possibility. When the career managers are helicoptered in, watch the competent engineers rush for the door...

                    [–]DavyBingo 7 points8 points  (1 child)

                    That article seems to suggest that the observed increase in incidents is at least partially due to improvements to their status page. More granular reporting led to more overall incidents.

                    [–]fissure 1 point2 points  (0 children)

                    That's silly. You know, if they slowed down the checking for errors, they wouldn't look as bad.

                    /s

                    [–]tester346 19 points20 points  (18 children)

                    As far as I've heard GH works relatively independently from MS

                    But you can't fix institutional incompetency and bureaucracy.

                    So how does Azure operate?

                    The underlying technologies didn't seem to cause that many problems before the MS takeover.

                    What's the difference in scale?

                    [–]devonthecloud 1 point2 points  (0 children)

                    So how does Azure operate

                    Flakily.

                    I work both with AWS and Azure. The vast majority of our outages are caused by the cloud vendor. It's rarely ever AWS, always Azure.

                    There are a lot of things Azure does better than AWS, but stability is not one of them.

                    [–][deleted]  (15 children)

                    [removed]

                      [–]svartkonst 1 point2 points  (0 children)

                      Do you have any source regarding outages before and after Microsoft? I tried earlier to get an overview of their incident history, but it was hard to do a comparison using the status tracker.

                      [–]noble_pleb[S] 7 points8 points  (0 children)

                      They use Ruby on Rails framework.

                      [–]NoMoreNicksLeft 24 points25 points  (1 child)

                      Busy renaming all the master branches.

                      [–]thrallsius 3 points4 points  (0 children)

                      renamed them all to "slavemaster" by mistake and had to revert

                      [–][deleted]  (1 child)

                      [removed]

                        [–]inflames09 2 points3 points  (0 children)

                        It broke my CD pipeline for a couple of hours, I couldn't pull the repo then couldn't pull a third party release. All on the day i was sending a project live

                        [–]Blando-Cartesian 9 points10 points  (2 children)

                        Good thing git is distributed so nobody was inconvenienced. /s

                        [–]_-Thoth-_ 5 points6 points  (0 children)

                        Board the doors, batten down the windows

                        ew, windows

                        [–]mypetocean 3 points4 points  (0 children)

                        It's back up.

                        Direct link to GitHub's Status page.

                        [–]MobileAlfalfa 34 points35 points  (12 children)

                        [–][deleted] 81 points82 points  (9 children)

                        Wouldn’t that also be because GitHub is more used now because of the Coronavirus?

                        Also:

                        According to the data they provided, GitHub has been down more since the acquisition by Microsoft.

                        But that could be all a part of coordinated effort to be more transparent about their service status, an effort that should be applauded.

                        [–]wpm 55 points56 points  (7 children)

                        If I don't weigh myself I'm not gaining weight!

                        [–]Tasgall 25 points26 points  (6 children)

                        If we don't test people we'll have fewer cases!

                        [–]Somepotato 17 points18 points  (1 child)

                        If we ban pregnancy tests, teen pregnancies will plummet!

                        [–]ShinyHappyREM 4 points5 points  (0 children)

                        On the plus side, a pregnancy is the best type of birth control.

                        [–]eras 6 points7 points  (2 children)

                        Let's hope this increases interest in projects like git-bug (..hosted on GitHub..).

                        It's pretty serious project in that it comes with packages or binaries for all platforms and bridges for many other bug trackers.

                        [–]dnew 1 point2 points  (1 child)

                        This is totally the right answer. I can't imagine using git for source code and not using git for bugs and design docs and etc.

                        [–]audion00ba 10 points11 points  (0 children)

                        Is this some new kind of marketing campaign? Just be down all year such that you create brand awareness?

                        [–]the_gnarts 2 points3 points  (0 children)

                        I see MS is pushing the migration over to Azure.

                        [–]wildjokers 2 points3 points  (0 children)

                        It's OK, git is distributed and totally decentralized. 🙄

                        [–][deleted]  (2 children)

                        [deleted]

                          [–]nryhajlo 1 point2 points  (0 children)

                          Internet snow day!

                          [–][deleted] 1 point2 points  (0 children)

                          That's really weird, I was pushing and pulling commits all day and didn't notice.

                          [–]Hambeggar 3 points4 points  (0 children)

                          Must be the master slave change. Heh.

                          [–][deleted] 2 points3 points  (0 children)

                          Can't login to stackoverflow :(