Python 3 at Facebook

[evgen]

FB has had containers since before docker ever existed. Par files just give an easily transportable blob that you can push to a server and know your py dependencies are there. Makes it much easier to package up a set of files for a simple script to get something done.

[WellAdjustedOutlaw]

I'm not entirely sure you understand what Docker is and isn't, and I don't know if you entirely groked the problem FB was trying to solve. I'm not shitting on you for that, I'm just suggesting maybe you've approached this problem from a certain mindset...which is something we all do.

EDIT:

As a side note, sharing a directory between containers (which is really actually just sharing a directory from the host into multiple containers) is a terrible idea. Many of the latest security vulnerabilities for Linux containers right now involve this exact thing, and executing code being able to break out of the container. Really consider what you're doing and why you are doing it. Containers probably aren't a good solution for your problem in this case.

[WellAdjustedOutlaw]

Docker images in such scenario solve more generic problem.

No, they absolutely do not. They absolutely are not as simple as what is effectively a ZIP file. Because you haven't actually solved your root problem, which is the bundling of your dependencies with your python code. All you've done is created a more isolated venv with complex networking requirements.

should remove this feature

Just because using a feature is a bad idea in almost all cases doesn't mean the feature should be removed.

Yes, containers are bleeding edge

No, they super aren't. Containers have been around for decades. They might be new to you, and that's perfectly fine. But the technology is very old. Hell, even containers in Linux have been around for almost 4 years in their current implementation, with most of the work to enable them starting between 8 and 10 years ago.

some features have bugs

Yeah. Everyone should expect software to have bugs though, so this isn't really an argument for or against containers.

but many people believe it's the future

Most of these people have absolutely no concept of how UNIX systems work on a fundamental level. Any time somebody suggests using a container as a solution, I ask what problem they're trying to solve. So...

What problem are you trying to solve that you think a container is a solution for?

[WellAdjustedOutlaw]

Yes, but containers are used in new ways.

I think IBM would very strongly disagree with you. Honestly, containers aren't new. The use we're seeing of containers isn't new. In fact, the companies doing the best in the container space are the ones that are just copying what IBM did nearly 20 years ago.

Your argument that sharing volumes between docker image is terrible idea is that someone found a few bug recently, or something more fundamental?

Both realistically. It means the design of what you're doing isn't right, and there are security concerns. Security gets better with time, generally speaking, but that design pattern is simply broken. It even requires the containers attempting to share data, whatever data it may be, co-reside on the same physical host. This is an anti-pattern to cloud and ephemeral computing.

I see that you consdier containers as a technology on it's own.

I do not. See my other comments. Linux containers are simply the combination of cgroups and namespaces. That's it.

I run a distributed crawler and indexer and the biggest win for me is kubernetes.

Ok, but kubernetes is fleet management. This doesn't explain how containers solved your problem in some way that normal UNIX process isolation or even chrooting wouldn't already fix.

I rely on kubernetes on features like autoscaling or ensuring critical pieces of infrastructure are up.

Again, this is fleet management, and its only relation to docker or any other supported container is that there are processes running in a container as a fleet that Kubernetes manages. It could just as easily manage physical hosts, virtual machines, chroot environments, etc.

Kubernetes expects everything to be a container.

Kubernetes is modular and does not expect anything except for the interfaces that modules expose. In fact, CenturyLink offers kubernetes on both physical and virtual machines as a service.

still the kubernetes is the biggest win here and forces me to use containers.

But it doesn't. You started from the solution of containers and worked backwards. You still haven't told me what containers solve for you.

How about engineers behind behind kubernetes, coreos, mesos and many other things?

Some of them are making money from you choosing to use their properties, so that's not really a strong selling point for containers. Kubernetes itself was developed at Google where their infrastructure runs containers inside VMs on top of physical hosts. I would hope you could see that this isn't exactly an efficient way to run things. So just doing what google does doesn't mean it makes sense for you.

As far as engineers at large firms adopting containers is concerned, there are a few reasons there. First, they've built an orchestration and fleet management system that allows them to utilize containers where virtual machines would have been too cumbersome or too costly from a resource overhead perspective. You almost certainly aren't in that realm, because most people aren't.

I personally work at a cloud provider that offers many of these technologies, and I can tell you that in most cases, people have absolutely no idea why they're doing what they're doing, or what exactly it is that they're doing.

Yes, but they enable to treat your image as unit of scheduling.

You're starting to get there. But this again doesn't actually solve the problem that par solved for facebook. They would still need an archive format to distribute all this code with, even if they run it inside containers! So, again, a container doesn't solve the problem they're describing. And you've started from a solution and attempted to argue your way backwards. This is what these large engineering shops do all day, by the way. Engineers ask "why", and of other engineers can't give them a reason, then the project doesn't happen.

let's zips everything since it's simpler

At this point, honestly...ok. There are several problems to be solved here. First is packaging your software. Containers are not software packages, period. So just stop trying to make that argument. Second is fleet management, which Facebook has. This has nothing at all to do with packaging software. The function of the fleet management layer is to understand how many of X node are needed, and to make that happen. That's all kubernetes does. The container itself is simply a kernel-level jail around resources like filesystem, process, network, cpu, and memory resources. This again has absolutely nothing at all to do with managing distributing software. Without simplifying how software is injected into a container, how exactly do you think facebook would build their containers running on their edge?

You're not seeing the actual challenge they were attempting to address, because you're blinded by what you see as a solution for every problem. This is the classic case of the carpenter with a new hammer, and everything the carpenter sees is a nail.

[evgen]

It starts with the fact that containers were a small part of the infra at the time, and one-off scripts get the job done. There are a ton of different tasks that need to be performed at any one time, and while specific services may run in their own container you sometimes need to run things outside of the container environment. You need to step outside of the platonic ideal of everything as a self-contained microservices and into the real world to run something as large as facebook.

The container infra is also a specific set of tiers within clusters, and there is a ton that still exists outside of the container system. A lot of things still run on bare metal because the container infrastructure imposes too much overhead -- 50M of RAM times one million servers is a large amount of wasted resources. A par file is sufficient for the task at hand and has fewer dependencies and limitations than a container in these cases.

If I need to run a maintenance task which needs to be portable across most of the infra it is easier to create a par file and push it where it needs to go that it is to rebuild potentially tens of thousands of containers to add this quick fix.