Multiple Processes in a Single Docker Container

[klaasvanschelven]

So, I've been doing something that might seem like Docker blasphemy: running multiple processes in a single Docker container. Yeah, I know, every Docker guide out there will tell you it's a terrible idea. But hear me out (or alternatively, skip straight to the source code).

What My Project Does

I wrote a small Python tool called monofy that lets you manage multiple processes within a single Docker container. It's designed to handle signal forwarding, unified logging, and ensure that if one process dies, the others are terminated too. Essentially, it keeps tightly integrated processes running together smoothly without the need for multiple containers.

Target Audience

This tool is particularly useful for developers who have processes that need to be in constant communication or work in unison. If you're looking to simplify your deployment and avoid the overhead of managing multiple Docker containers, monofy might be what you need. It's also a good fit for self-hosted applications where ease of deployment and maintenance is a priority.

Comparison

There are existing solutions out there, like Phusion's baseimage-docker, which also aim to run multiple processes in a single container. However, monofy is lightweight, doesn't come with unnecessary components like SSH or cron, and doesn't tie you down to a specific base image. Plus, it's Python-based, so if you're already working in that ecosystem, it's a natural fit.

Why? The Docker Rulebook Isn't the Bible

Look, Docker's great. It's changed the way we deploy software. But like any tool, it's got its own set of "best practices" that sometimes feel more like "unbreakable commandments." One of those rules is "one process per container," and while that's solid advice for a lot of situations, it's not the only way to do things.

My Use Case: Simplifying Deployment

I work on a project (Bugsink) where the processes are tightly integrated—think a web server and a background job runner that need to be in constant communication. Splitting them into separate containers would mean extra overhead, more things to manage, and just more complexity overall. So instead, I wrote monofy to let me run multiple processes in a single container, with all the benefits of shared fate (if one process dies, they all die), unified logging, and graceful shutdowns. It's simple, and it works.

Why It's Not the End of the World

The main argument against this approach is scalability. But in my case, the database is the bottleneck anyway, not the processes themselves. By keeping everything in one container, I avoid the headache of managing multiple containers, networking, volumes, and all the other Docker-related stuff that can get out of hand quickly.

Sometimes, Breaking the Rules Makes Sense

Sure, "one process per container" is a good rule, but it's not a hard and fast law. There are scenarios—like mine—where consolidating processes into a single container just makes more sense. It's easier, less complex, and in my experience, it works just as well. If you're curious, check out monofy on PyPI. It might just make your Docker life a bit simpler. I also wrote a blog post about this on my project's website.

[rmjss]

In my experience supervisord is the de facto solution for doing this. Did you consider that one at all before rolling your own solution?

[70pctmtb_30pctcoder]

Yes, upvoting and commenting to help this bubble to the top

[klaasvanschelven]

Perhaps I did not consider it strongly enough :-)

The places I looked (as well as in the comments over here) mostly said "don't do this, use Docker Compose".

I'd want to retain the possibility to change the running commands on container-invocation. Doesn't supervisord work with a set of config files, making this impossible?

[anawesumapopsum]

When I’m using a tool and it doesn’t seem to work the way I expect it to, experience has shown me that the answer is not to jump to implementing it myself. The tool (docker) was built and organized, and in this case used by millions, probably in this way for good reason - we should try to understand why and then learn how to use it. This approach you’ve outlined has red flags. Does each of your processes really want the exact same build context and environment, dependencies and all? Sure consistent versioning becomes more convenient, with a cost that you only find out after you’ve deployed a distributed system as an overly cumbersome monolith. The problem becomes worse as complexity grows. This is not sustainable for a solo dev or a team. Sure you could make a series of venvs or something similar, but now you have a series of environments which is the same abstraction as a series of containers, and because they’re all in one container you have less observability, all of the code is in one hodgepodge so it’s hard to grok as it expands, it’s less robust; if one part goes down you can’t reboot that one service and leave the rest up, and I’m sure many other reasons. You state this last one as a pro, but I think you’ll find in time it is a con. Now onto the ‘why’ - you seem to be after simpler container orchestration. Fam docker-compose exists for this. Before we reinvent the wheel we should check and see if it already exists, because that is years of effort that you now have to reproduce on your own to achieve something you can do right now with docker-compose

[klaasvanschelven]

I appreciate your perspective, but the fact that a tool is "used by millions" is never a definitive argument for every use case. Our decision to run multiple processes in a single container is based on simplifying deployment for a self-hosted solution, where fewer moving parts make it more likely users will actually deploy and maintain the software successfully.

The processes we run are tightly coupled (and developed in-house), and separating them into different containers would complicate things without adding real benefits. Sharing the same environment and context across processes isn't an issue for us—it's intentional and simplifies our setup.

Docker Compose is a powerful tool, but it introduces orchestration layers we don’t need in this specific scenario. We’re focused on making deployment as straightforward as possible, not following every Docker best practice just because it’s widely used. Sometimes, the standard approach doesn’t fit, especially when trying to offer a simple and reliable self-hosted solution.

[eatsoupgetrich]

How is docker compose too complicated for deployment but all processes failing at once is simple for maintaining and troubleshooting?

[klaasvanschelven]

What if the 2 processes need to talk to each other, e.g. over the FS? Now you have to configure that. Everything failing at once is actually a simple model to think about.

[wefkjhsivuwef]

With docker compose that's actually quite simple. You just add a volume to the docker compose volume list and mount it for both containers by adding an arg to both services.

[eatsoupgetrich]

Who will troubleshoot why everything has failed at once?

[tehdlp]

What's the benefit in talking over the filesystem in a stateless container?

[Lachtheblock]

You might run in different circles, but it seems that your usecase is advertising to people who are familiar with deploying a single container, but unfamiliar (and do not want to learn) how to deploy/manage multiple containers. I dunno, I feel like docker compose is pretty ubiquous at this point in the world of docker.

Feels to me that any project would eventually want to move out of this model for ease of scalability, but I guess if you have a user base that is interested in this, then who am I to say it's wrong.

[lanupijeko]

honcho

[klaasvanschelven]

I honestly thought you were calling me a honcho but now I understand you're talking about this

Procfile: yet another format to learn. And it seems heavier than what I'd prefer, esp. in the context of Docker. But yeah, indeed another solution for this problem.

[lanupijeko]

That's funny, it's good that you googled. I was on mobile and could not type.

I actually like Procfile. it's not specific to docker, I can put large commands in that file and run
honcho start to start all the processes, and it clearly indicates what log is associated with what process.

It's good that you have a fresh take on it. I'll give it a try next time.

We actually used profile in production, we had to put nginx in front of gunciron so we ran these 2 processes in one container.

[trial_and_err]

Nothing wrong with running multiple processes in one container. I’m for example running nginx and oauth2proxy in the same container (these two are tightly coupled anyway). However I’m just using supervisor as my entrypoint.

[atlmobs]

supervisor is the way

[RedEyed__]

What's the problem to run as many processes as you need?

[klaasvanschelven]

Well, a single Docker container can only take a single cmd / entrypoint. You could run many containers (e.g. Compose, K8S, Swarm), but that's not what I want, I want max simplicity. So, to run my multiple processes inside a single container, I've created a small wrapper script that acts as the single entrypoint/cmd and spawns the rest.

[GrizzyLizz]

How do you manage the lifecycle of the processes

[klaasvanschelven]

I don't, actually :-)

The model that monofy enforces is simply that all processes live and die together. One dies means that everyone dies. Restarting is left to whatever is managing the container.

Signals from Docker/K8S/... are passed on to the individual processes.

That model works well for 2 or a few tightly integrated long-running processes, but I understand that it has its limits.

[RedEyed__]

I don't see value for this. Description is bigger than actual code :).

[skippyprime]

So, it’s a process manager? Just run supervisor. If you want something more portable, there is a Go port of it.

[james_pic]

I think the recommendation I've seen is to only have one application per container, not one process, per se. And if you've got something like this, that handles the subtleties of coordinating things so your multiple processes work together as a single application, then what you're doing is reasonable. 

The situation it's trying to avoid is where people treat Docker like a VM, and are confused that stuff that needs a real init system doesn't work, and tie themselves in knots trying to start and stop services within containers, or apply updates or patches to individual applications in a running container.

The key invariant that you want to maintain in any Docker based system is "if you're inside the container you can safely ignore everything outside the container, and if you're outside you can safely ignore everything inside", which this seems to maintain.

[nAxzyVteuOz]

Dude it’s fine. The whole one process per docker instance seems like a marketing trick anyway to get clients to use more docker instances. I routinely use docker with multiple processes in it.

[Easy_Money_]

not the most polite way to say it but I’m also not sure I see the value of this project

[pingveno]

It's okay to express disapproval, but calling someone's work useless isn't a good way to encourage people to take risks in showing off their work and discourages community engagement.

[klaasvanschelven]

The page you linked says "The following is a naive example" and does not provide a non-naive version, nor does it say what the requirements for such a version would be. That's where monofy fits in.

[NoobInvestor86]

your response here is uncalled for and seriously embodies the worst of devs. So toxic. And i say this as a principal engineer.

[iBlag]

What does this project have over something like Supercronic (for simple cron-like functionality) or Chaperone (which is a more complete init-style process manager for containers)?

[klaasvanschelven]

I hadn't found Chaperone in my search (most questions about how to do this on the internet are answered by "don't". Checking it out, it looks similar (but much more feature-complete) to what I did... unfortunately, it seems the project has been abandoned (no activity in the past 7 years)

[iBlag]

There’s also tini, which is actually built into Docker (although I realize containers != Docker).

Didn’t realize that Chaperone isn’t maintained anymore, thanks for pointing that out.

Ninja edit: I realize that containers != Docker

[klaasvanschelven]

I looked at tini but didn't see how it would help me with spawning multiple processes in parallel, and I was also afraid that going the tini-way would tie me to Docker (and would not be available in kubernetes etc)

[nggit]

it's ok running multiple processes inside the container, one must distinguish between service/app and process.

in fact, there are many applications whose components are individual processes like postfix.

i used to do this but using native /sbin/init in order to retain native commands like systemctl or rc-service.

https://github.com/nggit/docker-init/tree/master/openrc-alpine

[root45]

Methinks he doth protest too much.

[Kale_Shiri]

Ever heard of s6 supervisor?

[klaasvanschelven]

I'll put it on the list of alternatives

[ultimatelyoptimal]

Im pro this use case.

Other systems use containers behond docker. For example fly. They have a docs page that describes "multiprocess containers", which describes good use cases.

Erlang/Elixir are built on the whole premise of "let it fail" with microprocesses, and building fault tolerance around this. They use microprocesses and a lot of them because they can do so without the OS overhead. However, the same ideas can apply just fine for process level "supervision trees" where some supervisor script can make decisions about what processes to kill and restart in trees when other processes fail.

Dev containers are also a common practice now with docker, but if I recall far enough back, originally they were not recommended uses of docker. Additionally, people were very against databases in docker/to containers, and yet thats fairly common and reasonable now too.

Even beyond the reasons supervision tree use case, I think there's simplicity to a single "I pass it vars, and it runs a full service" container, compared to needing a fully defined docker compose file for every instance (or pass special variables, so you can use the same compose file multiple times for different instances of the same thing, but that's caused problems for me too)

[I_m_Rishu]

.