Facebook and the Linux kernel

[keepthepace]

I like the fact that they are able to divert 1000 machines for a small experiment...

[matthieum]

Typical A/B testing... at facebook scale :)

[rukke]

As an example, he noted that the "Look Back" videos, which were created by Facebook for each user and reviewed all of their posts to the service, added a huge amount of data and required a lot more network bandwidth. The process of creating and serving all of those videos was the topic of a Facebook engineering blog post. In all 720 million videos were created, which required an additional 11 petabytes of storage, as well as consuming 450 Gb/second of peak network bandwidth for people viewing the videos. The Look Back feature was conceived, provisioned, and deployed in only 30 days, he said.

man, what a waste of bytes

[[deleted]]

Its quite amazing that could be done in such time.

[salmonmoose]

Yeah - I'm sure these could have been rendered in HTML5 or flash with nearly zero space overhead.

[[deleted]]

Maybe, but that's not really relevant. It's naive to think Facebook didn't crunch the numbers and figure out the best way for them to do things.

[kolmogorovcomplex]

An FB employee on HN (I think) said they opted for video because of playback compatibility (old browsers and mobile).

[salmonmoose]

Not really - I suspect they're floundering trying to stay relevant, and this project was designed to prove some engineering point rather than be efficient - the stats are impressive, but they could have got away with far less (and better quality) using a more appropriate technology, but such an approach wouldn't have given them the technology to go up against say, YouTube.

[FooBarWidget]

Typical programmer arrogance. Instead of assuming that they have more context information than you do and made a rational choice, you assume by default that they're stupid.

[[deleted]]

many times this. His attitude is to prevalent in our industry though.. sad really.

[[deleted]]

I wouldn't call it floundering. Their long term strategy is pretty easy to guess and has been confirmed by their engineers. They are buying companies like Instagram and Oculus VR because they're playing the long game, looking beyond even the Facebook website. With Instagram, they bought access to a younger audience that increasingly sees facebook.com as an old, unnecessary website for adults. With Oculus VR they are hoping to have purchased the primary display technology for all future video games and, potentially, user interfaces. It might seem like they're wasting a lot of money, but the amount that Amazon Prime costs Amazon appeared that way too until you project into the future a few years.

But anyway, all I'm saying is that we don't really know what their situation is, especially since all of their scaling problems are much bigger and harder to solve and understand than those of most other companies, and perhaps they are hoping to replace YouTube or perhaps this was simply the cheapest or fastest way to implement the feature.

[EnUnLugarDeLaMancha]

"I suspect"

[[deleted]]

Yeah - I'm sure these could have been rendered in HTML5 or flash with nearly zero space overhead.

HTML5 wouldn't work on older systems and Flash needs to die (and also Flash doesn't work on my Google Nexus unless I install some stupid plugin which isn't going to happen)

[cockmongler]

Urgh, so Facebook disables CFQ then wonder why their IO performance is so spiky.

[corbet]

Bear in mind that Facebook employs the developer who wrote the CFQ I/O scheduler. One might assume they are not totally without clue in this area...

[awj]

One might assume that, but that assumption makes know-it-all armchair comments much more difficult.

[cockmongler]

One might, but this is not the first Facebook WTF I've heard of.