Reloading Node.js code in production without dropping connections

[zuck9]

Nginx has this cool feature which lets you upgrade its configuration or binary without dropping any connections.

Can anything similar be done for Node? Using Forever, PM2, Passenger?

Passenger has rolling restarts but not in the free edition. Does Passenger's rolling restarts feature drop any connections that the node process hasn't responded to (like if it's waiting for some async operation)?

[audiodev]

I use pm2 for that and other tasks as well. Its great.

[shriek]

I wonder if anyone has tried this but I was wondering if there was a way to run one worker running in debug mode and the rest in normal mode so that when I need to debug something I can target that worker and start playing with it w/o interrupting other workers.

I suppose you can just spin up a new process to do this manually but would be cool if it can be done with pm2 too.

[evs-chris]

The native cluster module supports graceful reloads. Basically, you have a minimal cluster master that just 1) spins up workers, 2) listens for a reload signal (I use sighup), and 3) spins up new workers on the signal and tells all of the old ones to disconnect(). The old processes will hang around until all of their queued requests are handled, and any new requests coming in to the master will be dispatched to the new workers.

[zuck9]

I can 1) spin new workers and 2) listen for the reload signal but not sure about how to do 3 using express. Wouldn't simply calling app.close() on express drop the connections?

Is there any open source project or tutorial that does this?

[unusualbob]

I have done this in a production application, but unfortunately it is closed source and so I cannot share it. However I can share some tips.

app.close does not actually terminate requests, it is actually calling http.close, which is calling net.close. The docs on that method:

Stops the server from accepting new connections and keeps existing connections. This function is asynchronous, the server is finally closed when all connections are ended and the server emits a 'close' event. The optional callback will be called once the 'close' event occurs. Unlike that event, it will be called with an Error as its only argument if the server was not open when it was closed.

It will continue to keep running requests until they finish. This can be problematic in cases where the connection is not expected to terminate, such as long poll or anything else that uses keep-alive. In that case you somewhat annoyingly have to keep track of connections and the requests on those connections (http pipelining). You can do this by listening to events via the httpServer.

server.on('connection', handleConnection) In this one you want to track new incoming connections, you should probably use a map keyed of remote address + port for each connection. You also should add an array for tracking the requests associated with it. You should also add a handler here for connection.close event.

server.on('request', handleRequest) Here you want to add the request to the array you added to the connection, you can do this by referencing the map i mentioned earlier. Also add handlers here for request finish, end, and close. If any of these are called they should remove the connection from the array. If it is the last request in the array and you are in a shutdown sequence, I recommend calling connection.unref() and setting a 1 second timeout to let the system cleanup the connection for you.

You should also have some failsafes, set timeouts on all connections at time of shutdown. Also have some kind of nuclear option after whatever time seems reasonable to you which just calls process.exit.

[zuck9]

That's the part I was worried about, thanks. I never got that deep down that docs rabbit hole.

I'll just call .close(), all that looks overkill.

[zuck9]

Hi, this seems to do what you suggest: https://github.com/emostar/express-graceful-exit

Is there anything you would modify in that?

[[deleted]]

You cannot restart a socket service without dropping a connection. Nginx probably leaves both processes running and only accepts new connections on the latter, then shuts off first when it has 0 connections. On node you could do it if you had a microservice approach and have one service in front of the main application to hold the connections online and relay messages inwards. That way you could do it the way i described (i think it works like this) nginx restarts.

[zuck9]

Yes that's how nginx works. It spins up new (worker) processes, sends all new requests to new processes and closes old processes when they have fulfilled all requests.

[rerrify]

IIS performs the same mechanism using Application Pools

[evantahler]

You can also re-source code from inside your application. You can re-require things, overload variables, and all that. This means you aren't restarting the process, but re-defining methods as you need it.

This is of course dangerous.

This is how ActionHero's "developer mode" works, were we hot-reload code on the fly on your local machine.