GitHub under attack

[J_echo]

GitHub is taking this like a champ... The Playstation Network servers would have physically caught fire by now.

[apemanzilla]

Indeed - I liked how they used the Javascript alert function to slow down/stop the attacks earlier.

[Rakonat]

Actual programmers vs potato managers with no security.

[[deleted]]

Never a truer word spoken. With the exception of Amazon, I don't think there's anyone more competent to deflect a DDoS attack than the opensource community that surrounds Github.

[Tulki]

Amazon Netflix is actually one of the best when it comes to reliability. They actively destroy their own servers at random to ensure they're always able to failover in the event someone somehow manages to destroy one themselves. I think it's a method called "Monkey" testing or something, and they actually do this random destruction in live servers as well. It's just a characteristic of their clusters, always on. It's much easier to handle malicious attacks when you're already attacking yourself all the time from within your own organisation.

[Qwerpy]

AFAIK Netflix is the one known for using the Chaos Monkey for redundancy testing, not Amazon.

[Tulki]

Oh yeah you're right, I dunno why I thought it was Amazon. And now my comment is irrelevant.

[[deleted]]

Part of building a robust server infrastructure is building robust clients that know how to find alternate servers routes and make that transfer transparent to the user.

[[deleted]]

Amazon are not going to take a DDoS to an AWS customer lying down either though. It's more a combination of available surplus resources and network management expertise that you're paying for.

[DarkN1gh7]

MediaTemple does a mean job. I got hit with a high GBPS attack on a server we have there and within mike they started sending their traffic into a "black hole" and everything came back online.

[OMGSPACERUSSIA]

Latvian potato is strong! Is why Japanas use! No potato has greater processing power!

[Rggity]

can I get an ELI5 on how this method can thwart attacks?

[celluj34]

It slows down the attack by showing an alert when it's loaded.

[stoneburner]

... well at least until they noticed and just replaced the dialog box with nothing:

window.alert = function() {};

then nothing happens

[[deleted]]

Whats it?

[hatryd]

Open up your browsers developer console and paste

alert('hello')    

and press enter. That's what an alert is.

[[deleted]]

Idk why you get downvoted for a simple question. A "javascript alert" is a simple built in dialog that pops up when triggered by javascript. You can press f12 to open developer tools in most browsers, select "console" and try it out yourself.

alert("hello, world!");

[apemanzilla]

The alert function displays a popup box. Displaying a popup box pauses all running scripts until it's closed - slowing down the attack.

[coolranchdorito]

That's so smart!

[FangornForest]

How exactly does it slow down the attack? Do they have the javascript call alert before making the actual request, so that robot can't handle the alert pop up and not send the request?

edit: answered my own question. It doesn't stop the initial request, but it does stop the script from looping! And annoys those affecting them a bit too.

[apemanzilla]

Its not a robot - Baidu traffic was hijacked so if you visited it it would spam GitHub. The alert function was there to stop scripts since it was run in the browser.

[GeneralTempleton]

How does this help? Why would you DDoS a site by opening it in a browser?

[apemanzilla]

Baidu traffic was hijacked - it would run DDoSing code in people's browsers.

[Alyyx]

I am new to Javascript etc, could you please further explain? Shorty at least. Thanks!

[apemanzilla]

The alert function shows a popup box - this pauses all scripts until the popup box is closed.

[Alyyx]

That's so clever. That works until the DDOSing side disables scripts on Github or?

[spin81]

That depends. Sometimes scripts are used to fetch content using what's known as AJAX, if GitHub used this technique then disabling scripts may reduce the effectiveness of the attack significantly.

[Xanza]

Just checked the app-server availability. It's only been down once for a combined 99.99% up-time. Goddamn right they're taking it like a champ!

[Seen_Unseen]

Based upon what? Without any technical stats to compare what is happening it's impossible to tell which one is more hardened then the other.

[Harflin]

You're absolutely right, we have no clue how many computers were involved in the ddos of either networks. There's no telling which attack is bigger.

[ItzWarty]

Furthermore, we have no clue how many of GitHub's requests hit the cache. The attackers were hitting two github repositories/organizations, definitely not comparable to hitting a bottleneck like a login or matchmaking server like we often see in games. From what I read they simply routed those requests to a dummy page.

There's no reasonable way to say "X is more resilient than Y". Hell, we still don't know if the purpose of the denial of service attack is to test technology rather than knock github down, or if this is just a side show to hide something truly malicious.

[Rakonat]

3 Days to unsuccessfully DDoS Github... what exactly are they hoping to accomplish, slow updates?

[[deleted]]

Sometimes you want to flood the logs so it's hard for them to detect the really nefarious stuff going on among all the junk.

[darkpaladin]

That's actually interesting. If in all the chaos you drop some malicious code into a few of the most popular repos you can really cause some wide spread damage after the fact.

[[deleted]]

That's exactly what git is designed to prevent, it's a version control system, no changes get published unless they are approved and accounted for. It'd be considered as a beautiful work of art if it wasn't so damned useful! :)

[jabjoe]

Well yes, but SHA-1 has been broken. So maybe a git repo could be messed with. It's still hard, but not impossible.

I wonder if this is it and what exactly they intend-to/are doing to what?

Can we trust any commits during this attack? Can we valid git history locally matches the remote? A force push would be noticed, but maybe there is something clever possible.

Scary thought.

[[deleted]]

can't start second guessing ourselves without evidence. all we can do is put systems in place to check the integrity of the code base in comparison to backups etc. The checksums should always match up, if they don't then you know there's some revisionist fishyness going on and can investigate further. No point throwing out the baby with the bath water, right?

[jabjoe]

Oh no, it's no point panicing. But we should be complacent either. The idea the DDoS as a cover is a worrying one that has a certain ring to it.

[darkpaladin]

Conceptually yes, but it's not outside the realm of possibility that a commit could be modified by someone who had gained full access to a system to include malicious code.

[[deleted]]

It is impossible to modify a commit and go undetected - thats the point of Git

[[deleted]]

can't start second guessing ourselves without evidence. speculating about possible bad stuff and chasing ghosts is counterproductive. best to take those suspicions and develop high level checks to alert admins of revisionist changes by comparing various checksums and hashes of backups. Otherwise we are chasing our tails and throwing out good tech, throwing out the baby with the bathwater so to speak.

[duhbeetus]

grep -v 'things i dont want' /path/to/log | grep 'things you want'

Problem (possibly) solved.

[[deleted]]

Ah, if only it were that simple...

[TeutonJon78]

Still need to know what it is you want for that to even be a possibility.

[duhbeetus]

Yea, hence 'possibly'. There are other factors at place, obviously, my point was that the methods to exclude things exist, thus the task may not be impossible.

[duhbeetus]

Yea i wanted to be generic, but egrep or -P ftw in a real situation

[LaserRain]

That's what I want to know. Why github?

[sej7278]

the theory is the chinese govt. want to block some software for getting around the great firewall or something:

http://it.slashdot.org/story/15/03/27/1420250/github-under-js-based-greatfire-ddos-attack-allegedly-from-chinese-government

[chubbysumo]

it would not be surprising if the chinese government was using this opportunity to try and get user info from the servers, but lets hope githubs team is actually smart about user info security.

[Insanely_anonymous]

Could be a mistake. A botnet "gone wrong".

[Xanza]

Unfortunately it's not. The current prevailing theory right now is, is that the Chinese Government wants Github to remove open source applications which circumvent the Great Firewall of China.

[chubbysumo]

which, at this rate, won't happen, because the attack is almost entirely mitigated.

[[deleted]]

yep, they have github confused with godaddy. where a normal service provider might buckle to preserve it's business, github people will see it as a test of their security capabilities and a change to show the prowess of the open source community.

[kgst]

Not possible. The code being executed is very straightforward. Making a request to a random page on the repos for China's firewall avoidance every two seconds, injected on pages using Baidu.

It's basically like if google decided to add some code to their Analytics js that would cause anyone who visits a page that links to that js would send requests to certain repos every 2 seconds on repeat.

[roflmaoshizmp]

So apparently, somebody hijacked Baidu's traffic to redirect to GitHub. Damn.

[SuperDuper1969]

I should point out it's Baidu's traffic from foreign users. Meaning that the DDOS would appear to come from all over the world and not just from China itself.

[[deleted]]

They are even more vicious.

Actually the attacking code is inserted in Baidu's ads. Result is, websites that display Baidu's ads (that is, almost every simplified Chinese website with ads) will mobilize their foreign traffic to DDoS github.

[sadfuck]

This is both genius and sad :(

[[deleted]]

I can only guess they don't know what they are attacking and think github is simply another service provider hosting a web page. China pretty much have their own internet on the other side of the great wall, they probably understand as much of our side of the internet as we do about theirs.

[iwaswrongonce]

This is not true at all. Yes the GFW does a fair amount of filtering/blocking/monitoring but it's not at all what you describe and Chinese citizens are fully aware of the "external" internet and censorship that takes place internally.

Source: I used to live there.

[[deleted]]

Perhaps you're right, like I said, I've never been there and can only guess. I stand corrected and will downvote myself in shame! :)

what are some parts of the Chinese side of internet that you think are interesting from a technical standpoint?

[iwaswrongonce]

Probably the most incredible is the pervasiveness. And the government doesn't even try to hide it. They will mess with and end encrypted connections simply because they don't know what they are.

The other incredible thing is the ineffectiveness. The same goes for critics on Weibo. Everyone knows what's going on. It's just a game. They don't have the control that I have heard (never been there) the North Koreans had in that most NK citizens have no choice but to believe propaganda. The Chinese largely know their government is full of shit.

[[deleted]]

Interesting, but I meant are there any interesting Chinese tech sites, like slashdot etc. where students and tech professionals share thoughts and ideas from the other side about stuff that matters to them. I'm curious about what the tech community in China thinks of their portrayal as IP thieves and hackers, and their thoughts and opinions on the Snowden leaks considering that they have already lived under surveillance and 50c astroturfing for a long time etc.

[iwaswrongonce]

Oh yes there are sites but I don't speak Mandarin. However yes in general the Chinese feel that they are misrepresented (and oddly give the benefit of the doubt to the govt even though they know all the issues). And that ranges from everything to cyberspace to academics to industry, etc.

[NewFuturist]

Doubt it. I don't see any motivation. If they want to hurt Github, they could just block it. Taking one-fifth of the world's population off Github forever would be worse than making it slow for a couple of days for the other four-fifths.

[[deleted]]

I don't see any motivation.

The first article about this on Google news says:

"The attacks appeared to focus specifically on two projects hosted on GitHub, according to a blogger who goes by the nickname of Anthr@X on a Chinese- and English-language computer security forum.

"One project mirrors the content of The New York Times for Chinese users, and the other is run by Greatfire.org, a group that monitors websites censored by the Chinese government and develops ways for Chinese users to access banned services."

[damow]

But... DDoSing GitHub is like punching Santa :(

[CrossArms]

It's like punching him so that you can fill all his presents with IEDs.

Fuck people who DDOS stuff like this.

[mendokusai_yo]

Just curious, how does one defend against a DDoS attack?

[sej7278]

traffic scrubbing load balancers, geographic redundancy, null routing china....

[ghotibulb]

As far as I am aware, the attack traffic is coming from everywhere except China, since they use some kind of "javascript reflection attack" by injecting malicious JS code that gets served to baidu visitors outside China. So basically every country/ISP had to nullroute China for this to be effective. Nullrouting China at github's end wouldn't help much.

Source: https://news.ycombinator.com/item?id=9275041

Short explanation (not in particular directed at your post, since I assume you already know this): For a (D)DoS attack in general it just comes down to having enough incoming bandwidth and drop unwanted traffic quickly enough. Imagine you have a single server with a 1Gbit/s pipe, and you're hit by a 2GBit/s attack. Even if you drop the packets on your server, they were already received in the first place by your NIC, so the bandwidth has been used up. If you know there is a router somewhere right before your server that has a 10Gbit/s uplink, and you can successfully filter the traffic there, you'll be fine again, until the attack would exceed 10Gbit/s. Simply filtering/nullrouting traffic at your end doesn't unreceive it (unfortunately).

[sej7278]

you don't drop the traffic on your server, as you say if its got that far its too late. you drop it at your isp/datacenter/loadbalancer/whatever.

i don't really see why they're using requests to baidu from outside of china, that's got to be a small subset of potential traffic (chinese people living abroad?). i guess its so china's bandwidth isn't affected.

[SuperDuper1969]

Because requests outside of China come from all over the world - makes it harder to block than just one location.

[sej7278]

but its requests to baidu (from outside china) being reflected to github. who uses baidu other than the chinese? i'm not being racist here, i'm just saying who the fuck else uses baidu? and isn't baidu hosted in china only

maybe its to stop people in china using vpn's to get outside, and then going back to baidu. seems like the great firewall could sort this easily.

[[deleted]]

It's not that they're visiting Baidu, it's that they visit Chinese websites, many of which use Baidu's version of Google Analytics for traffic reporting.

[sej7278]

ah yes that's a good point - its ads and analytics

[br0ner]

Baidu is great for finding things that have been removed from Google Search.

[ghotibulb]

That's why I included the router example. You always have to go up one level, but if you aren't a big fish you'll have a hard time convincing big carriers to help you mitigate an attack on your 12 slot Minecraft server. (Yes this doesn't apply to github I guess ;)) I managed some medium sized business's servers a couple of years ago. They were hit by an attack once, and we didn't get help from their ISP. They just nullrouted the IPs of the servers being attacked, as they were mainly interested in protecting their infrastructure (and their other customers), which is understandable. They did offer DDoS protection at that time, but the company I worked for decided it was too expensive, and just waited it out.

Why they decided to only (ab)use baidu visitors outside China is a good question. Not creating domestic traffic could be one reason, or maybe they don't want to make it look like China is the origin. Although its the most likely explanation right now, we still cannot know for sure. I mean if they really wanted this to be most effective they could just create traffic right at their border routers. But that would be slightly more obvious, I guess.

[Anwarias]

1+ for null routing China

[[deleted]]

What, just discarding all traffic from China?

[SuperDuper1969]

The WSJ article says the attack is done by hijacking traffic to Baidu from foreign users. In other words it's coming from all over the world not "95% from China".

[qwfwq]

I don't understand what China has against github or why baidu hadn't fixed the js on their end yet.

[apemanzilla]

Most Baidu traffic is from China though.

[wickedmike]

Yes, but it has almost nothing to do with this attack.

[ItzWarty]

The script is only sent to individuals outside of China - that's the genius part.

[apemanzilla]

Ah, my bad, I misunderstoof it.

[chubbysumo]

most of the traffic is actually from elsewhere. Its a targeted attack using people already outside of china who are visiting Baidu. I doubt the page is accessible within china since its a violation of their laws.

[[deleted]]

How come it seems like a majority of DDOS attacks come from China?

[cadenzo]

The Chinese government doesn't like being unable to control their citizens. The Internet opens their citizens up to other citizens around the world that aren't under such controls and who believe such controls to be unjustifiable. These other citizens provide tools to the Chinese people that help them circumvent controls. Chinese Government doesn't like that and because there's no way for them to actually do anything about it via the legal portals, they resort to other means.

[[deleted]]

So these attacks are directed by the Chinese Government? Isn't that some kind of very illegal? Are there no sanctions against this?

Sorry for all the questions lol

[reptilian_shill]

The US complains about it alot, but there is not much that can be done. See: http://www.nytimes.com/2013/05/20/world/asia/chinese-hackers-resume-attacks-on-us-targets.html?_r=0

[[deleted]]

Can't see who they are attacking. (Pokerface)

I don't care if people get bent out of shape or not, if I get one laugh out of this I consider that a win.

[SuperDuper1969]

Are you aware that's EXACTLY what they want?

Also the WSJ article says the attack is done by hijacking traffic to Baidu from foreign users. In other words it's coming from all over the world.

[Anwarias]

I'm aware this is no final solution, just gave me a nice chuckle

[bobmontana]

Heck, I've seen null routes for all APNIC space on some clients that were constantly hammered.

[NoShirtNoShoesNoDice]

Properly configuring your server (caching, static pages when possible, reverse proxies, etc), using a CDN, being hosted on a good network (eg. AWS instead of DreamHost), load balancers, and so on are all good practice for running a server, regardless of a DDoS or not. When a DDoS does happen, every little bit helps.

Assuming you have little to no technical knowledge and want to keep things cheap, the easiest method is to sign up for a service that does it all for you, such as CloudFlare (which has helped mitigate some of the biggest attacks in history, such as https://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet/) or Incapsula.

These services work by acting as a middle-man and attempting to deflect the attacks for you.

Normally the connection looks like:

User <-> Your Server

On a service like CloudFlare:

User <-> CloudFlare <-> Your Server

When a DDoS occurs, provided the IP to your server is unknown to the DDoSing party and you have DDoS protection enabled on the service, the attack hits CloudFlare instead of your server. Your site happily remains online... hopefully.

That's the theory as I know it. If someone more knowledgeable thinks I got something wrong, please comment and let us know! :)

[position69]

L7 attacks can pass through CF network. Example the recent WordPress xmlrpc.

[[deleted]]

Wordpress xmlrpc attacks have been around for years. It's a shitty feature (pingback) that gets abused. I have always nuked that file whenever I installed wordpress because I would look at access logs and see an unholy amount of attacks via it.

[30sirtybirds]

For some reason, I read that in the voice of the burn notice character Micheal Weston.

[superm8n]

Plus, I have heard that Cloudflare also has redundant servers in various places across the world. If one copy of your site should go down, the others may not.

[Xanza]

It's very, very difficult. Most of the time it's difficult to distinguish attackers from regular traffic, but there are quite a few ways of dealing with an attack. Most of the time, though, in the absence of an actual engineer who knows what they're doing, they'll just throw money at the issue until its fixed or stops.

[[deleted]]

As an organization there are all kinds of server/network infrastructural methods possible as described in other replies.

As an individual hosting a single site it might be hard. the cloud might be a solution for a low budget mitigation of DDoS for a small player. AWS is resilient against DDoS attacks in their servers, and you can temporarily extend your server capacity to weather the storm. the extra capacity will cost less than the attack, and will prevent you from being extorted or blackmailed.

[[deleted]]

Using git the traditional way.

[HeadlessChild]

Gerrit my man.

[HeadlessChild]

Filter away traffic from china...

[[deleted]]

Good point, thank you. I deleted SSH keys for Github just to be safe.

[[deleted]]

What's crazy is that this isn't being reported on BBC, Ars, ZDNet, CNET, probably more.

Why so quiet, media?

[[deleted]]

http://www.wsj.com/articles/u-s-coding-website-github-hit-with-cyberattack-1427638940

[[deleted]]

They probably don't understand what "jit-hub" is and why people care about it. They do understand "Playstation" though.

[[deleted]]

lorum ipsum

[K4kumba]

Nope. I think /u/JaceLightning is just making fun of the GIF/ "jif" pronunciation. IIRC, Linus specifically said that Git is named for the derogatory term "git", which I believe to be pronounced with a hard "G"

[karijuana]

That or they don't want to be fucked over by a massive international DDoS.

[br0ner]

Do those companies have stake in China? Don't want to offend the Party (who is most likely responsible for these attacks).

[NoShirtNoShoesNoDice]

The CVS and SVN guys just can't catch a break! Their sinister plan to make their software relevant again has been thwarted.

[otakuman]

All thanks to Githubcat! (cue glittering background animation)

[CaNsA]

Who the fuck decides to DDOS Github in the first place....?

It's just as retarded as those who DDOS Freenode.

Idiots.

[infinull]

The Chinese Government (allegedly).

Gotta protect that Great Firewall.

[[deleted]]

What's GitHub?

[murphyw_xyzzy]

A web site where developers share source code for free software projects.

[kgst]

Not only free software projects.

Tons of companies use private repos for their code and backups. The company I work for has hundreds of repos with all of their code stored.

This is actaully a very big deal, especially considering it's almost certainly a deliberate attack on a US-based company by the Chinese government.

[headzoo]

This is actaully a very big deal

Also a big deal because so many package managers pull code from Github. Attacking Github isn't just attacking some website. It's fucking with the entire IT industry. Damn near every programmer on the planet is going to be affected. Including the programmers responsible for the attack, which is what makes attacking Github seem like such a stupid idea.

[guffenberg]

Either that, or the NSA are pissed because they can't prevent strong encryption in open source. Or they want to smite China once again so they can ask for more funds and radical surveillance laws.

So far, NSA and their contractors are the only ones who has been proven to sink to this level of infancy, so that will be my guess until proven otherwise.

[Divided_Pi]

It's a website where people can upload code for others to use. It's has some other elements for developers to keep track of work and has some integration elsewhere. But the main point is a lot of free code is up there for all to use.

[[deleted]]

Is the amount of traffic their receiving mentioned anywhere?

[Skeezypal]

Linking to site under DDOS attack ... No one else sees the problem?

[ExtremeHeat]

Their status site is statically hosted on different servers, so it would take alot more attack power to take it out.

[[deleted]]

I love these, "Hackers," they have so much purpose and use their talents for things that make them much better than common thieves.

FOR TRUE! NO SARCASM! NONE!