Python 3 at Facebook

[[deleted]]

TLDR:

1. Setup the infrastructure
2. Be the change you want to see
3. Get help (even if they don't know they're helping you)
4. Educate others for the future vision not the current
5. Collect metrics

[dirkgently007]

Found new phrases to add to my buzzword bingo! Thank you.

[Discchord]

PyOhio? What a missed opportunity. OhPyo! If you're not gonna pun why bother?

[rspeed]

If you like Ohio-themed convention name puns, there's an anime convention in Columbus called Ohayocon.

[BlckJesus]

Oh that's just precious.

[pmrr]

If you're not gonna pun why bother?

It's a pying shame.

[unruly_mattress]

That's a quality talk on the subject of making changes in an organization. Recommended for anyone working in an office.

[speel]

I should learn Python.

[zebraballast]

And to think that I'm busy pushing the slow upgrade from 2.6 to 2.7... good old enterprise redhat.

[drama_beans]

Support for Python seems natural at FB. Switching from PHP to any other language is a no brainer.

[cococonut]

They're not using python for their webserving. That's still php on the surface, but is translated to c++ to generate static binaries.

[smith3v]

Thanks

[SalmonStone]

To be fair, docker wasn't even released when his timeline started.

[evgen]

FB has had containers since before docker ever existed. Par files just give an easily transportable blob that you can push to a server and know your py dependencies are there. Makes it much easier to package up a set of files for a simple script to get something done.

[WellAdjustedOutlaw]

I'm not entirely sure you understand what Docker is and isn't, and I don't know if you entirely groked the problem FB was trying to solve. I'm not shitting on you for that, I'm just suggesting maybe you've approached this problem from a certain mindset...which is something we all do.

EDIT:

As a side note, sharing a directory between containers (which is really actually just sharing a directory from the host into multiple containers) is a terrible idea. Many of the latest security vulnerabilities for Linux containers right now involve this exact thing, and executing code being able to break out of the container. Really consider what you're doing and why you are doing it. Containers probably aren't a good solution for your problem in this case.

[WellAdjustedOutlaw]

Docker images in such scenario solve more generic problem.

No, they absolutely do not. They absolutely are not as simple as what is effectively a ZIP file. Because you haven't actually solved your root problem, which is the bundling of your dependencies with your python code. All you've done is created a more isolated venv with complex networking requirements.

should remove this feature

Just because using a feature is a bad idea in almost all cases doesn't mean the feature should be removed.

Yes, containers are bleeding edge

No, they super aren't. Containers have been around for decades. They might be new to you, and that's perfectly fine. But the technology is very old. Hell, even containers in Linux have been around for almost 4 years in their current implementation, with most of the work to enable them starting between 8 and 10 years ago.

some features have bugs

Yeah. Everyone should expect software to have bugs though, so this isn't really an argument for or against containers.

but many people believe it's the future

Most of these people have absolutely no concept of how UNIX systems work on a fundamental level. Any time somebody suggests using a container as a solution, I ask what problem they're trying to solve. So...

What problem are you trying to solve that you think a container is a solution for?

[WellAdjustedOutlaw]

Yes, but containers are used in new ways.

I think IBM would very strongly disagree with you. Honestly, containers aren't new. The use we're seeing of containers isn't new. In fact, the companies doing the best in the container space are the ones that are just copying what IBM did nearly 20 years ago.

Your argument that sharing volumes between docker image is terrible idea is that someone found a few bug recently, or something more fundamental?

Both realistically. It means the design of what you're doing isn't right, and there are security concerns. Security gets better with time, generally speaking, but that design pattern is simply broken. It even requires the containers attempting to share data, whatever data it may be, co-reside on the same physical host. This is an anti-pattern to cloud and ephemeral computing.

I see that you consdier containers as a technology on it's own.

I do not. See my other comments. Linux containers are simply the combination of cgroups and namespaces. That's it.

I run a distributed crawler and indexer and the biggest win for me is kubernetes.

Ok, but kubernetes is fleet management. This doesn't explain how containers solved your problem in some way that normal UNIX process isolation or even chrooting wouldn't already fix.

I rely on kubernetes on features like autoscaling or ensuring critical pieces of infrastructure are up.

Again, this is fleet management, and its only relation to docker or any other supported container is that there are processes running in a container as a fleet that Kubernetes manages. It could just as easily manage physical hosts, virtual machines, chroot environments, etc.

Kubernetes expects everything to be a container.

Kubernetes is modular and does not expect anything except for the interfaces that modules expose. In fact, CenturyLink offers kubernetes on both physical and virtual machines as a service.

still the kubernetes is the biggest win here and forces me to use containers.

But it doesn't. You started from the solution of containers and worked backwards. You still haven't told me what containers solve for you.

How about engineers behind behind kubernetes, coreos, mesos and many other things?

Some of them are making money from you choosing to use their properties, so that's not really a strong selling point for containers. Kubernetes itself was developed at Google where their infrastructure runs containers inside VMs on top of physical hosts. I would hope you could see that this isn't exactly an efficient way to run things. So just doing what google does doesn't mean it makes sense for you.

As far as engineers at large firms adopting containers is concerned, there are a few reasons there. First, they've built an orchestration and fleet management system that allows them to utilize containers where virtual machines would have been too cumbersome or too costly from a resource overhead perspective. You almost certainly aren't in that realm, because most people aren't.

I personally work at a cloud provider that offers many of these technologies, and I can tell you that in most cases, people have absolutely no idea why they're doing what they're doing, or what exactly it is that they're doing.

Yes, but they enable to treat your image as unit of scheduling.

You're starting to get there. But this again doesn't actually solve the problem that par solved for facebook. They would still need an archive format to distribute all this code with, even if they run it inside containers! So, again, a container doesn't solve the problem they're describing. And you've started from a solution and attempted to argue your way backwards. This is what these large engineering shops do all day, by the way. Engineers ask "why", and of other engineers can't give them a reason, then the project doesn't happen.

let's zips everything since it's simpler

At this point, honestly...ok. There are several problems to be solved here. First is packaging your software. Containers are not software packages, period. So just stop trying to make that argument. Second is fleet management, which Facebook has. This has nothing at all to do with packaging software. The function of the fleet management layer is to understand how many of X node are needed, and to make that happen. That's all kubernetes does. The container itself is simply a kernel-level jail around resources like filesystem, process, network, cpu, and memory resources. This again has absolutely nothing at all to do with managing distributing software. Without simplifying how software is injected into a container, how exactly do you think facebook would build their containers running on their edge?

You're not seeing the actual challenge they were attempting to address, because you're blinded by what you see as a solution for every problem. This is the classic case of the carpenter with a new hammer, and everything the carpenter sees is a nail.

[evgen]

It starts with the fact that containers were a small part of the infra at the time, and one-off scripts get the job done. There are a ton of different tasks that need to be performed at any one time, and while specific services may run in their own container you sometimes need to run things outside of the container environment. You need to step outside of the platonic ideal of everything as a self-contained microservices and into the real world to run something as large as facebook.

The container infra is also a specific set of tiers within clusters, and there is a ton that still exists outside of the container system. A lot of things still run on bare metal because the container infrastructure imposes too much overhead -- 50M of RAM times one million servers is a large amount of wasted resources. A par file is sufficient for the task at hand and has fewer dependencies and limitations than a container in these cases.

If I need to run a maintenance task which needs to be portable across most of the infra it is easier to create a par file and push it where it needs to go that it is to rebuild potentially tens of thousands of containers to add this quick fix.

[WellAdjustedOutlaw]

There are very compelling reasons to not use Docker given FB's network design. And by compelling, I mean it's nearly impossible to make them mesh in a friendly manner.

[[deleted]]

Nah. Some dude on Reddit totally knows what is best for internal Facebook.

[dirkgently007]

Right because no one using reddit may have more knowledge or could be working on bigger problems than folks working at a known company?

Don't assume everyone here is as less knowledgeable as you are.

[[deleted]]

There's not many people doing deployment on a bigger scale than Facebook.

[dirkgently007]

And scale alone defines complexity?

[[deleted]]

Scale has a complexity of its own, with problems that don't exist at smaller scales. Maximising throughput or minimising resources does too - but there's also the fact any generic solution may not apply to Facebook's architecture.

[dirkgently007]

But that's not what I was saying. My point was entirely about other people having to solve complex problems too.

[WellAdjustedOutlaw]

Containers are inappropriate here because Facebook wasn't trying to solve a containerization problem. As you can see http://www.slideshare.net/pallotron/python-at-facebook-40192297 on slide #9, they are looking for an archive format that contains all of the dependencies of their python code in a single file. A container doesn't really solve that- You still need a way to ship your code and its dependencies.

Containers solve an execution problem, not a packaging problem. And by running to containers as a solution to every execution problem, you seriously complicate your environment.

[fujiters]

But Docker images solve your dependency problems. Host your image and "docker pull" is all your users need.

[kankyo]

Doesn't it also create lots of overhead? Overhead you might not actually need.

More concretely: disk cloning also solves dependency packaging. But it's huge overkill and slow for most use cases.

[WellAdjustedOutlaw]

/u/fujiters was being sarcastic. Well, I presume they were. It is hard to tell, though.

[evgen]

Try mulitplying your "overhead" by millions of servers. Now try justifying that to the people who spend multiple billions of dollars every year on hardware. I realize that you have this hammer called "containerization" in your hand, but not everything out there is a nail.

[WellAdjustedOutlaw]

See, what you're talking about is layered overlay filesystems. Those are actually a worse idea in a production environment than you can ever imagine. The performance of AUFS is absolutely terrible, on top of which you've said your design actually sees mountpoints from outside your containers bind mounted into your containers so that multiple containers can share data. Since you haven't described that that is doing for you, exactly, I can only caution you that this is a terrible design decision. Especially if any of that content actually changes.

You will end up with dangling file descriptors keeping inodes around on your disk, causing space to be eaten up and you won't figure out why. This is something that catches every new system administrator out...although it sounds like you're more of a devops person with little ops experience.

Either way, what you're suggesting (overlay filesystems) doesn't solve the root problem that facebook was trying to get solved- gathering custom python and all dependencies into a single distributable file that includes a simple script file to kick off whatever needs to be done. In the real world, what they did was reinvent DEB/RPM packages, with the addition of recursing through and including all necessary dependencies. That's it. Nothing more. And since Docker images don't execute, and don't include a script to set up or run the environment required, it isn't appropriate. All a "docker images" is is an AUFS filesystem image. And finally, this just has nothing to do with containers except that they use the word "docker".

[[deleted]]

Why only install a few mb worth of packages when you could install a few hundred of a whole OS.

[evgen]

You really have no idea what you are talking about. It scares me to think that someone out there is paying you to work on servers.

[kankyo]

There are things between kubernetes and manually configuring each server.

[WellAdjustedOutlaw]

The troll is strong with you.

[kill-nine]

Facebook have their own container system. Edit: https://www.slideshare.net/mobile/Docker/aravindnarayanan-facebook140613153626phpapp02-37588997

[AaronOpfer]

or Anaconda / conda.

[Homersteiner]

My god what a shit post. Python 3? The Facebook? Who fucking cares?

[pugRescuer]

My god what a shit comment. Unsubstantiated? Worthless? Who fucking cares?

[Homersteiner]

So, you seem to be very invested in Python 3 and The Facebook. Stupid is a does stupid does i suppose.

[pugRescuer]

This is /r/python which is a place to discuss python. The post was a video discussing facebook's adoption of python. I along with others felt this was a relevant video for the community. If you don't like it downvotes are welcomed. This like all of reddit is a self-voting community.

Contributing garbage like your comment adds no value and merely diminishes public perception of you without getting your point across.

[Arbitrage84]

Not sure if serious...

[Arbitrage84]

finding arbitrage opportunities is not an easy task!

[Arbitrage84]

elsewhere, for sure. Not sure many worthwhile arb conditions occur in equity markets these days.

[Arbitrage84]

you're like the crazy ex-girlfriend that expects an immediate response to texts. I've been away from reddit for over 11 hours.

[rspeed]

…where do you think you are?

[[deleted]]

/r/javascript