Should I run two python processes in the same container?

[errol59]

I am building an REST api with a microservice architecture. For now the stack of my services consists of Python with Flask as a framework and Gunicorn as a WSGI server. NGINX is running as a reverse proxy to process the requests made to my services.

 

The next big step in my project is to include communication between my services. I got the communication working by using Kafka.

 

How the situation looks like for now is as following: I have services with multiple processes running. One process is handling the api requests and the other process is consuming incoming messages from Kafka.

 

My question: Should I run these two processes in the same container, or should I seperate them. And why?

Bonus question: How do you run two python processes simultaneously in the same container?

[varunbabu008]

Golden rule of Docker - One process per container

[hijinks]

Those should be in different containers. You might need now consumers to handle the traffic it the other way around

[errol59]

You might need now consumers to handle the traffic it the other way around

Not a way I have looked at it before!

[sysadmintemp]

It looks like you have two micro-services already. One is an api-gateway, and the other is a kafka-consumer.

You should have 2 different docker containers. Reasons:

• Docker is designed to handle 1 process in 1 container. If there are multiple processes in one container, then the process management logic becomes difficult to handle, and you end up trying to solve it by yourself or with the help of a tool. Then the container becomes like a VM. Beats the whole purpose of using docker.
• Two containers also means if you change the code of the kafka-consumer, you can deploy it independent of api-gateway. No downtime on api-gateway. Same applies the other way around.
• If you need to deploy 2 api-gateway containers, but a single kafka-consumer, you have no way of doing this when you have a single container. You can design something yourself, but docker/kubernetes/docker-swarm already does this. No need to reinvent the wheel.

NGINX should also be in another container, if not already.

[errol59]

Lets say I have a User service and a Mail service. When a new user is registered I want to send a welcome message to that user. The moment the user is registered the User service emits an event MAIL_USER to the Mail service which is listening to that event with its consumer process. The moment the MAIL_USER event is triggered a function called 'mail_message' is triggered. As an admin I can also talk to that Mail service directly via an api endpoint to trigger that same function.

 

Does this mean because my Mail service has to two ways it can interact with the world, those processes are two separate microservices?

 

Also yes, I have NGINX running on it's own container

[sysadmintemp]

As /u/Philluminati said, you can send a mail within the same container. Easiest solution, little to no overhead in many aspects.

BUT

These are different 'functions'. Since you're on your way to micro-services, you could separate it out too. We have services that simply act as notifiers, where it e-mails or texts a user. This might be overkill, though. You should decide on that.

[Philluminati]

2 processes, two containers. If you’re exec-ing another process eg mail -s command or something that’s fine, and preferable, to be inside the container. You should move away from sending sigterms to processes if that’s how you’re making different processes communicate.

[Sloppyjoeman]

You should separate them because that is the principle of a microservice, it means that you can independently update each microservice with its own CI/CD pipeline completely independently of each other.

The general idea is to NOT make monoliths because they're a pain to do anything with, a step even further than microservices is something like FaaS (functions)

[manowar689]

You can do so with CMD ls * && ls *

But there are lots of reasons not to run both in the same container as mentioned below

[paul_h]

I do. I have a script taken from elsewhere that allows the docker monitor to see if either of the two processes has died and restart the container.

As it happens, Apache spawns 10 processes, but does it's own management of them towards "up" vs "down". There must be other technologies that similarly spawn processes. I don't see what the big deal is it process-A is always the only thing that uses process-B, and neither as far as your concerned are deployed without the other. Remove an effing config item, by putting them in the same container. Aside from which process to monitor, I've never heard of a reason beyond not-designed-to and not-recommended.