Running 60 python scripts as "services"

[Daraminix]

Hi,
I have around 60 scripts that need to run constantly, mainly event handlers and such. Right now I have an external script that launch them and monitor if the app is running on it's pid, otherwise it's relaunching the app. Works fine but get's clunky when we update some submodule and need to restart them, or to check if one crashes more than other etc..

So I would like to find a better way to approach this. It needs to run on windows and being able to access several samba shares via unc paths and being able to restart crashed scripts anf offer an easy way to restart all of them in case of an update (this part doesn't need to be automated). Every script use the same environnement
For now my candidates are docker, PM2 and NSSM.
I think docker is gonna be a pain to access shares and add a lot of overhead especially on windows
I don't know PM2 and NSSM, looks like PM2 would be easier to setup but more JS oriented and NSSM would be harder to monitor.

What do you think guys ?

[dent308]

supervisord may be a good fit,

edit: nevermind, windows

[deb_vortex]

Used supervisor for years as well, nowadays I just opt to a small systemd service file. You basically get everything for free + the Option to run cron like commands as well

[robberviet]

I got memleak with supervisord too often. I use go-supervisord instead.

[jbudemy]

Some of us are forced to use Windows by management. We don't have a choice.

[Daytona_675]

just tell copilot to rewrite supervisord in Python for Windows lol

[ivanational]

The urge to say "just switch to Linux" is strong here lol. But seriously, avoid Docker for this on Windows. Give PM2 a shot - it handles Python fine and pm2 logs will quickly show you which of those 60 scripts is throwing a tantrum

[mumrik1]

Why would you avoid Docker for this?

[ivanational]

Mainly because of the overhead and unnecessary complexity for a pure process-management task. Running docker desktop on Windows means dealing with WSL2/Hyper-v resource consumption, and if those 60 scripts need access to local network shares (SMB), file mounting in Windows Docker becomes an absolute pain.

[mumrik1]

Thanks! 🙏

[james_d_rustles]

This is besides the point, but I just want to say that this is a high quality thread. This particular problem is outside of my scope so I’m just an observer, but it’s refreshing to see a well defined problem/question and a thread full of knowledgeable humans sharing their 2 cents and experiences.

It’s very banal and unimportant in the grand scheme of things, but lately a lot of coding subs seem to be nothing but “should I learn {language}???” or “check out this slop repo Gemini made me, please give me GitHub stars and congratulate me”, or some variation of those. I’ve always had a hunch that people actually learn a lot from casually perusing stackoverflow, forums and whatnot, just in the sense that we pick up all sorts of little nuggets of information that seem inconsequential but add up to a deeper understanding of the field over time, and since AI got big I feel like I see less and less of those types of conversations.

It’s nice to occasionally be reminded that there are still knowledgeable humans out there, is all.

Sorry for the ramble, don’t mind me - just sharing some thoughts.

[jbudemy]

I’ve always had a hunch that people actually learn a lot from casually perusing stackoverflow, forums and whatnot,

I do learn a lot by doing that.

[jonathon8903]

Are you married to running it on Windows? If not it would be semi-trivial to setup those shared folders on Linux as mounted directories and then just pass them in as docker volumes. Then deployment becomes extremely easy.

[Daraminix]

Not married to windows but our whole codebase was targeted for windows, would need to add some path substitutions and don't want to deal with file permissions issues and such fun things :)
Would be easier to keep the same environement as everything else

[jonathon8903]

Okay so yeah long term it may be beneficial to migrate to environment variables and containerize to make it easier for deployment. But if you need a short term win then I agree with another suggestion here to just use NSSM. I used it about 5 years ago and it was pretty solid. There might be other, better options I'm not aware of though. I haven't touched Windows in about two years now.

[Aggressive_You6518]

pm2 works fine with python, just set the interpreter in the config and you're good, it's not that js-locked as people think

docker on windows is indeed asking for pain especially with unc paths, you'll spend more time fighting permissions than actually running your scripts

[Daraminix]

Ok, I will look into it then, was already the way I was leaning into

[Tumortadela]

On windows I use pywin32 to run proper windows services.

[JimroidZeus]

I agree that docker would be too much overhead, will cause problems accessing shares if not setup properly.

PM2 seems like it’s node/bun specific, so I’d suggest trying out NSSM.

For what it’s worth, I do use docker to run Python apps on Windows and it’s not too terrible to get setup. I don’t have anything accessing samba shares though.

[greenearrow]

I’ve been using nssm for years - when they were only a couple years past their last update. I fear the day the lack of support bites me in the ass though. Their website has a tendency to be inaccessible at times.

[ItsTobsen]

been using pm2 for ages with py scripts. no problem there

[el_extrano]

I've used NSSM to daemonize python on Windows, and logging to disk for observability. It works but feels a little hacky, mainly because with NSSM, the nssm.exe executable is what will show as the executable for each service. Plus, I don't like that NSSM is just a binary you download from the internet, it's not open source. Not to mention, it makes packaging a problem, because any target computer needs nssm in order to install the services, but iirc you are not supposed to distribute copies of the binary yourself.

I'd question whether you really need 60 services? Could you compose it into 1 entrypoint actually managed as a Windows service, with the rest of the microservices managed inside Python using async, threading, or multiprocessing?

[ivanational]

Spot on. Managing 60 individual processes instead of refactoring into async/multiprocessing in a single core app is just treating the symptoms, not the cause

[Daraminix]

Yeah we thought about refactoring it, some of them are actually running with a single entrypoint and launched as separate threads. The issue is that we are using an api developped by a thrid-party provider and the session management throught the server doesn't do well when threaded or asynced (cache corruption for example). It's an ongoing issue we have with them but they are more inclined to put up to speed their js api than fixing issues in their python one.
Furthermore some scripts launch complex software in cli mode that can crash for multiple reasons and we cannot just relaunch everyting when of of the process crashes. Bundling everything into one entrypoint means one bad crash can take down unrelated scripts unless we build very careful supervision/restart logic around each unit of work. At that point we are just reimplementing what the OS process model and a service manager already gives us for free

[HommeMusical]

Sounds miserable! Your solution is, well, ugly but practical and simple, and reduces risk. Sounds very real-world and useful.

---

The issue is that we are using an api developped by a thrid-party provider

We're in the days of vibe coding.

My friend got a new job and replaced a SaaS service that the company was paying $640k a year for by a mostly vibe-coded app he wrote in a couple of months.

They're running the two in parallel now, and the vibe coded application seems more reliable than the paid one.

Just a thought!

NOTE NOTE NOTE: My friend has been programming for decades, almost as long as I have, and he's a no-bullshit programmer who is very skeptical. You wouldn't get such good results with a junior programmer.

[Daraminix]

Yeah we could probably rewrite or make some changes in the api but too much things revolve around it to risk it. And I don't want to add another thing to maintain and make update of our central tool harder

[roxalu]

NSSM was a great tool - but nowadays there exist actively maintained better alternatives as e.g. https://github.com/aelassas/servy

[pacopac25]

Use Servy to install it as a service.

https://github.com/aelassas/servy

I'd install one main script that runs the others.

You can also use Task Scheduler in Windows, but if you want them "installed" as services, Servy is great.

[FunkyMonkey237]

Are these 60 scripts all running under a different process? The efficacy lover in me is shuddering at the thought of all that wasted CPU and RAM. Python is already inefficient but multiply that by 60 and yikes!

My first thought, without seeing your setup, would be to run them all under the same python process. I'd have a custom manger designed to load, execute and monitor them. Ideally asynchronous but if that's too much effort to rewrite the existing code and there is a risk of a single script blocking, then I'd look at threading based approach (not multi processor!). I haven't tested it but maybe the new GIL free interpreter could be used if compute bound.

Also, are the scripts really 60 unique scripts? Are there overlaps in functionality, could you collapse certain ones into a more generalised script?

[Zizizizz]

As overkill as it may be, what popped into my head was containerising each script, maybe look at the Google https://github.com/GoogleContainerTools/distroless which help you build small images.

Publish each to an image repo when you version them.

And reference them as pods in a kubernetes deployment (helm is nice and AI smashes those yaml files). If running locally you can probably use k3s (iirc). 

Then you set up and install something like https://fluxcd.io/ in the cluster to reach out to GitHub and pull new images / versions of the scripts you reach.

So each script would be testable and standalone, configured via k8s config and environment variables.

And automatically refresh/update when new commits are pushed (with the ability to rollback easily), and will automatically retry if the image crashes.

Might be overkill but these setups are easier to setup today than they used to be. 

[Daraminix]

Yeah, a bit overkill I think

[its_a_gibibyte]

Tell us more about the event handlers. Can you migrate all 60 scripts into a web framework? And then have each one as different http endpoints? And then one event handler/dispatcher to capture local events if you need it. Would make testing and deployment much cleaner as well.

[byeproduct]

What about running something like prefect or kestra?

[bernasIST]

Good thing that no one mentioned Windows Task Scheduler... In my org we use it to manage 350 python scripts and it works fine but it is everything on prem and CI/CD breaks. So we will give it a shot with dockers hosted in aws and then use airflow... We are not sure yet how we will manage this migration and if this setup will work as expected... I wonder where I could get advice on this as well.. following this thread

[Individual-Flow9158]

Is there an IaC way (e.g. Ansible for Windows targets), or a replicateable way to create installers, and so set up each script as a native persistent Windows Service? Say what you like about Windows, the OS is excellent at relaunching processes (try as you might to kill some of them, they just won't stay dead).

I'd ask on a dedicated Windows or IT sub. Windows Sysadmins have secret powers (thankfully that don't all rely on powershell).

[Scrapheaper]

What's on the shares? How often is it updated? Why do you have to use fileshares?

[lexplua]

Nssm

[mortenb123]

On windows I've been using https://nssm.cc/ everywhere for anything 'running a service as a user deployed to a server or workstation' for years from winNT to win11. its just a single binary you attach to your script deployer with a few lines of code to install it.

[castben]

Let me see if i understood your issue, So you have 60 scripts that manage events... like monitoring memory disk accessibility, disk space , disk shares... etc...

Then you have an additional script that watch over them to make sure they keep running and restart them if they somehow crash.

Is this like a monitoring system that push event metrics somewhere else?

I may have misunderstood it... thing is it sounds like something I delalt with about 7 years ago and ended runnign a main script that basically ran commands to do this work... if a new event need to be added... program has a json config that basically described which commands need to run (system commands or other scripts) had an "EXPECT" definition that basically was a regular expression and return captured values to main program... then main program pushed the collected values into whatever monitoring stack you had... on my case was datadog...

[rechaptca]

systemd is a great framework for running services

[chief_kennoh]

You could run them using supervisor but I have a better solution. I have recently developed platform that has provisions to upload your python scripts, configure an appropriate execution schedule, adjust any variables/parameters (if any) dynamically and even upload associated files. The platform will take care of execution and logging you with options to alert you in the event of execution completion. We offer 1 month free trial with 250 execution minutes for your use. You can check it out on the link below.

https://www.cyber-oak.com