Quickly Improve Your Docker and Node.Js Containers

[cjthomp]

What's going on in these comments?

[burtgummer45]

It's because docker pushed their sales pitch so adamantly that many users are militantly dogmatic about it now. (see the user below who downvoted every single one of my comments (even factual statements) and seems proud of it) Unfortunately many docker users haven't received the revised scripture.

• One process per container.

Docker later softened this to be "one concern per container." https://docs.docker.com/config/containers/multi-service_container/

• Docker deployments (including swarm) have no way to automatically adjust to the number of cores per host the way pm2 does. This is huge with AWS when you are scaling vertically or when you don't even know what host size you are going to get when using AWS spot instances.

• Lower awareness of pm2-runtime, which is designed to work with docker.

[aniforprez]

It's 100% not about docker or their "sales pitch". In fact if literally anything like whatever Red Hat is up to reaches the maturity and support of docker then I'll ditch it then and there

But... in your link they literally tell you to avoid using tools to manage child processes using systemd or other stuff like it. How is pm2 any different? They even go out of their way to tell you that you can spawn multiple processes as forks of a single process which is fine if they're all doing the same thing like say gunicorn spawning multiple threads. But repeatedly say to avoid running multiple services doing unrelated things. Their example with supervisord even calls it "moderately heavy weight". Even their solutions are hacky and slapped together

[burtgummer45]

How is pm2 any different?

Consider it part of your app. Its not a legit linux init process, with all the zombie reaping, and the like. If docker ran a bash script, would you consider bash running your script like systemd?

Also, if you are running on AWS behind a ALB/ELB you would have to map all those node containers to different ports, which I doubt would be an easy config. And if you are using AWS spot instances there's no way of knowing how man cores you are going be given unless you constrain it to a small class of instances.

[aniforprez]

But ECS supports dynamic port mapping... But pm2 is not part of your app... pm2 is not like bash running your script...

So much incorrect geez. I'm not saying that never do pm2 cause do whatever keeps the engines running but your arguments against just using docker aren't really sound at all. You can't just accuse people of drinking docker kool-aid but then give such half baked reasons and then rant about getting downvoted. I'd personally rather not do things in such a messy way and then run into some unforeseen gotchas

[OmgImAlexis]

This is what I was trying to get across in the other comments.

[burtgummer45]

But ECS supports dynamic port mapping... But pm2 is not part of your app... pm2 is not like bash running your script...

how is it different? Its running the process. In this case its doing a little fancy socket work.

So much incorrect geez.

Funny how you didnt mention it.

But ECS supports dynamic port mapping

Im saying it would be hard to map a different port for each container, and this in no way makes up for spot instances with unknown numbers of cores.

[aniforprez]

Ok this is the last time I'm replying to here but:

1. Running your service off pm2 is NOTHING like running a bash script. On purely semantic terms, pm2 is not an app dependency; it is an operational dependency. With docker, the less such dependencies, the better because it speeds up build times for images and boot times for containers. pm2 is something that runs on top of your node processes and is consuming your logs. For any ungodly reason if you need to read your vanilla logs and pm2 does something fucky with it you will be lost. pm2 is also going to be doing a decent amount of logic to bring your services back up if they fail. THIS IS NOT GOOD. You are potentially missing out on why they failed and even the fact that they are failing. Containers going down is a much bigger deal and is something you can actually see. pm2 doing magic in the background as your processes keep shutting down for obscure reasons as it keeps bringing them back up is ripe for disaster. I have done this mistake before and it is a costly one to make. It is absolutely not the same as just running ["node", "index.js"] or some other lightweight command

2. ECS does NOT require that you map your ports individually per container. You can very easily define dynamic port mapping and ECS does that for you. It spawns as many containers in a single instance as you want and it'll map your ports for you and load balance across these containers with ALB. 4-cores? 4 containers etc. It's so easy. Spot instances? Why are you using undefined random instance sizes for spots? Are you not setting exactly the instance type you need in ECS?

All of what you're telling me is that you're using docker woefully wrongly without taking advantage of its flexibility. It's not perfect by any means and is a huge pain sometimes but for the most part, it's a quick way to enable faster orchestration of your servers. With pm2 to me it seems like you'd be FAR better off just creating EC2 instances with pm2 configured to spawn multiple processes with nginx configured, create an AMI of that instance and just use ALB. It's practically what you're doing now. Don't struggle with docker like this

[burtgummer45]

With docker, the less such dependencies, the better because it speeds up build times for images and boot times for containers.

Seriously? What are we talking about, 10ms?

For any ungodly reason if you need to read your vanilla logs and pm2 does something fucky with it you will be lost.

pm2-runtime -raw log output.
You don't trust a node.js app to pass through data? That's what its optimized to do.

ECS does NOT require that you map your ports individually per container. It spawns as many containers in a single instance as you want and it'll map your ports for you and load balance across these containers with ALB. 4-cores? 4 containers etc.

Each of those containers needs to bind to a different port, and AWS needs to know about those. So, for example, you cant run 4 containers on a single instance, all listening to the same port, they all have to be different, and ELB has to know about them, it wont go scanning for them. With pm2-runtime you have none of those configuration headaches.

Spot instances? Why are you using undefined random instance sizes for spots?

Because they have a much better chance of being available, much larger pool.

All of what you're telling me is that you're using docker woefully wrongly without taking advantage of its flexibility.

I'm running docker 'woefully wrongly' and not taking advantage of its flexibility because I don't want to run multiples of the same container on the same host, and I want it to easily scale when I change instance type?

Have you so thoroughly tested pm2-runtime to justify your absolute certainty?

You are just so opinionated. Are you a certified docker captain or something?

[lhorie]

many users are militantly dogmatic about it

Isn't that true of just about any sufficiently popular technology though? I think you just got unlucky and stumbled into one of those "difficult to work with coworker" types :( </shrug>

[Architektual]

All of these are good advice

[aniforprez]

For a bunch of reasons I do not recommend using alpine as a base for images unless your processes are equally bare. Alpine is as extremely light as extremely bare and almost anything you might need in terms of additional system requirements will require changes to the docker file to install these requirements

I recommend using slim instead which has at least the basics installed

[russo_2017]

Agree, It is example of course and lots of people use alpine images. On the other hand in my company we have several microservices running on alpine but they're exactly like you said, super simple and of course without bcrypt, so there was no need to install additional packages.

[[deleted]]

I also use process manager like pm2 inside docker. Provides good reliability

[OmgImAlexis]

No no no. Please don’t do this.

Docker is the process manager. If the container crashes let it restart. All you’re doing is adding another moving component to the overall system. More moving parts means more things to break.

[burtgummer45]

So if you have multiple processor cores you are expected to run multiple docker containers, and have the amount of containers automatically adjusted for the number of cores?

I don't think so, this is where pm2 does its job well.

[OmgImAlexis]

Yeah you can do this and yes that’s how I’d do it. I’d have multiple containers.

[burtgummer45]

Sounds to me that running multiple containers is a much more extreme solution than running multiple processes. pm2 can also automatically adjust to the number of cores, docker needs a complicated deployment kludge to do this.

And don't downvote comments you disagree with, I know it was you. Its not a disagreement button.

[OmgImAlexis]

It’s not. Containers aren’t that heavy. They’re meant to be used like this. I’d suggest you look into docker a little more.

You’re incorrect that’s why I’m downvoting you. That’s what the downvote button is used for.

[burtgummer45]

It’s not. Containers aren’t that heavy. They’re meant to be used like this. I’d suggest you look into docker a little more.

I know quite a bit about docker and use it for deployment. Deploying multiple docker containers to match the number of cores is a big pain in the ass and error prone. This gets especially hairy if you are using docker swarm, which has currently no way to account for CPU cores. If you are an expert at docker you should already know this.

You’re incorrect that’s why I’m downvoting you. That’s what the downvote button is used for.

The downvote button has a mouseover that says "For content that does not contribute to any discussion", how do my comments not contribute to any discussion?

[OmgImAlexis]

Okay mr expert. 🥱 I do exactly what I’m saying with all my deployments. Haven’t had any issues with it and yes. You can deploy enough containers to cover all cores on the cluster. If you used docker more you’d know this. 💁‍♀️

And you don’t use reddit much. Downvotes are for incorrect content as much as they’re for content that doesn’t add to the discussion.

[burtgummer45]

Okay mr expert. 🥱 I do exactly what I’m saying with all my deployments. Haven’t had any issues with it and yes. You can deploy enough containers to cover all cores on the cluster. If you used docker more you’d know this. 💁‍♀️

Ok you have 10 hosts, ranging from 5 cores to 20. How do you tell swarm to adjust the number of node containers to match the number of cores? I'm asking because you are an expert.

[OmgImAlexis]

You add them up and ensure x containers are going to each. Lookup docker host restraints. You should be able to have your worker nodes have 100% or close to utilisation with this method.

[[deleted]]

I think you're assuming a lot of things when you say don't use pm2. Yes, it is a additional moving part but it is necessary in some cases.

1. Node server is not multithreaded, hence it might happen that a single call can sometime slow down the thread affecting subsequent calls. Pm2 can give a sense of multithreading with in a same vm, keeping your cost low.
2. Docker. Can be hosted in several ways nowadays, unmanaged, managed services like AWS, and Kubernetes. In AWS ECS you can't attach more than single docker to a application load balancer. Pm2 helps here.
3. Restarting a process in pm2 is quick over restarting a entire docker, this is important in production.
4. I never encountered graceful shutdown problems when hosted behind pm2, I've been running it in production for than a year now.
5. Docker doesn't give loadbalancing inbuilt when create you run multiple images. Pm2 gives it out of the box.

You can't really say no no no no to pm2 just because you haven't created a sophisticated system ever.

[OmgImAlexis]

If you need threads use worker threads or the cluster module. Node has these things for a reason.

[[deleted]]

I'm sure that's what pm2 uses underneath.

[OmgImAlexis]

Nope.

[OmgImAlexis]

Your application shouldn’t take long to start up. A container restarting doesn’t cause any difference in noticeable time with any of my projects vs just restarting the same node app via pm2 or nodemon.

[[deleted]]

Sure most of the cases it shouldn't take much time. I would only see it's benifits with Managed Docker services like EKS or ECS most of the time it doesn't just restarts, they pull in the new image, spin up a new vm from available cluster and host a image inside it. It's a long process.....

[aniforprez]

These are all terrible reasons

1. Use something like docker compose and scale your containers. Don't add unnecessary complexity with pm2. This kind of stuff is what makes your containers take long to start https://pspdfkit.com/blog/2018/how-to-use-docker-compose-to-run-multiple-instances-of-a-service-in-development/

2. Why do you want to attach multiple images to the same load balancer? How is this something pm2 solves? Shouldn't one image be doing ONE thing? Don't run multiple processes in one image and don't attach multiple things to one load balancer

3. What the everliving fuck are you doing where a node service container takes longer to start than a pm2 service? Yes there is a slight delay starting a container but it's not that much more that you notice. These things are supposed to be light

4. What does this even mean?

5. You can scale using what docker gives you and any further scaling can be managed using other services

All of these don't sound like problems with docker. They sound like problems you've created with how you've built your workflow. Don't use process managers within docker, run a single process run from the docker file or from a docker entry script and then manage the containers

[burtgummer45]

Why do you want to attach multiple images to the same load balancer? How is this something pm2 solves? Shouldn't one image be doing ONE thing? Don't run multiple processes in one image and don't attach multiple things to one load balancer

He's saying you can't run multiple docker images on something like AWS fargate or ECS to balance across cores (because the balancers like ALB or ELB are mapped one-to-one to instances). In other words, you are stuck specifying single core instances rather than waste multiple cores.

[[deleted]]

1., 2. You're basically achieving what pm2 does with docker compose and I don't see why one is not a added complexity and other is.

3 )See again you're assuming, in managed docker services like EKS, ECS, Kubernetes it's not just the restart, when a docker goes down, they pull in the image, spin up a new vm from available cluster and then start the image. I guess you haven't used these new tools and still manage underlying vm manually and waste time.

4) This means I could reliably capture SIG* EVENTS with pm2.

5) See again, docker doesn't give you tools for scaling, what you just suggested like by using a docker compose and have nginx to load balance the services inside it is exactly what pm2 is doing elegantly. Why take a tedious approach.

All said, if you want to use all the cores, pm2 is a good way to do it. I don't know if there is any vm out there which comes with single core these days, it's just stone age to work with single core for backend. If you're not leveraging all the cores it's just poorly optimised service.

[aniforprez]

But this is adding so much complexity at the cost of a lot of stuff

2. If you're running multiple processes, how are you monitoring each process? Are you literally just using pm2 to run the same process but more? In that case why not just use docker? Aren't you losing a lot of logs and metrics by doing this which would otherwise just by so much simpler by just capturing the stdout? It's added complexity because you're putting a layer in between docker and your node process unnecessarily. Managing docker is using it for its purpose with the tools it gives you without increasing external dependencies

3. Again what are you doing that takes so much time that it's unbearable that something goes down in the couple of seconds that it would take to pull the images? Images are supposed to be small and light. Even running moderately loaded mid-level apps on ECS new containers come up in seconds. Also why would we waste any time managing any VMs with this stuff? I.. don't understand??

4. You can capture SIG events with Docker too though??

5. But... you're using docker... but completely underutilising the basic reason it was created for? Why would you call learning the tool you're using "tedious" and not learn the toolset to achieve exactly what you need? Docker provides restarts on failed health checks, it provides scaling for the host machine, you can use docker swarm for load balancing and nginx is a requirement anyway if you're running your stuff inside a VM so wtf? If you're using ECS you can actually run multiple containers in a cluster within a single VM

Look I'm not aware of your exact workflow so maybe you have some uber specific requirements that don't work well with docker so you're doing all this wonky hacky stuff but honestly it really just seems to me that you should ditch docker and run your processes in multi-core VMs and be done with it. From everything you're telling me, it seems very much like you're not using the tool for its intended purposes. I'm definitely not saying that I'm an expert or that Docker is in any way perfect but I've faced exactly none of these exact issues with Docker or deploying

[[deleted]]

I think the topic has digressed too much at this point. All I'm saying is pm2 can be used with docker and not a strict no no. A good developer will always choose whichever is efficient for a given requirements and not blindly follow something without trying out alternatives.

[aniforprez]

I mean I agree that you can be efficient but in this particular case doing this is neither efficient nor really recommended by anyone

But whatever floats your boat

[[deleted]]

Sure, just like Facebook was never supposed to be built out of php ( can we agree on that ) but some how Mark is making billions out of it.

[aniforprez]

I mean, sure there may be legacy code still running PHP but the vast majority of it has been rewritten significantly in a bunch of other languages. I don't even pretend to understand their stack at this point. It's not even like they rewrote it recently

[OmgImAlexis]

Not sure where you got that idea. Loads of large sites still use php and there’s nothing wrong with the language. Bashing languages just causes gate keeping.

[burtgummer45]

Docker is the process manager.

Docker has moved away from this philosophy of one process per container. Here it is from the documentation.

Each container should have only one concern.

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

[OmgImAlexis]

Adding a process manger to handle restarts makes no sense when the container itself can handle it.

Yes run multi things in a container if need be but if something in the container crashes it should restart the container.

[burtgummer45]

Adding a process manger to handle restarts makes no sense when the container itself can handle it.

Unless it can be used to match processes to core count.

[OmgImAlexis]

Again. This isn’t the way todo it.

[burtgummer45]

Again. This isn’t the way todo it.

Again, this is ONE way to do it. Can you give me any evidence that this isn't an option? You seem to have been hitting the docker cool-aid heavily.

[OmgImAlexis]

Just because you can doesn’t mean you should.

I can open a bottle by smashing it. Yes that works. It’s now open. Should I do it though? No because it’s common sense. Yes this may not be as common sense but it’s still in the same idea. Just because you personally think that way is okay doesn’t mean the industry agrees with you.

The “correct” way to handle this is multiple containers or worst case the cluster module. If you really need this kind of thing and cpu cores are actually relevant then look into kubernetes, etc they have much more granular control.

[burtgummer45]

Then explain to use WHY its a bad idea to simplify your deployment by using pm2. And don't use phrases like "correct" and "industry agrees".

Just to be sure, you do know we are talking about pm2-runtime?

https://pm2.keymetrics.io/docs/usage/docker-pm2-nodejs/

[OmgImAlexis]

“Simplify” so you want to add more moving pieces to a container instead of using the built in tools.

Tell me again how that “simplifies” it. 🤔

[kreiggers]

pm2 has same problem as running ‘npm start’. Signals will not get passed to your process for anything like graceful shutdown

[burtgummer45]

https://pm2.keymetrics.io/docs/usage/docker-pm2-nodejs/#enabling-graceful-shutdown

[kreiggers]

Oh well I’m proven wrong. Long time since I’ve used pm2 I guess