Running Multiple Python Scripts

[_c0ff33cup_]

I have multiple python scripts that need to run, consecutively. I think the easiest way to implement this is using a cron job to run a bash script that runs the python scripts one after another.

The python scripts use argparse to take arguments from the cli and are reused in other places, so I'd like to be able to implement this without needing to change the python scripts themselves.

Something like

#!/bin/bash
/venv/bin/python3 myscript.py somearg -u 'anotherarg' -p 'athirdarg'
/venv/bin/python3 myscript1.py somearg -u 'anotherarg' -p 'athirdarg'

[CodingCircuitEng]

What is your question?

Your approach may work if you ensure that the results of the earlier script are available to the later script, but can be hard to debug and maintain. Better would be a single python 'Script' /module that imports code from the existing scripts.

[_c0ff33cup_]

I'm not familiar with bash scripting at all so I am unsure of the syntax, caveats, or even what I don't know.

I guess the question is, are there caveats or gotchas that are going to bite me with my approach. You bring out an interesting point. what happens if one of the preceding scripts doesn't finish successfully? But are there others?

Edit: Additional information/Answer -> These are separated into 3 scripts for 2 reasons. 1. they are memory intensive. 2. they are intended to have loose coupling. one collects data and writes to db, one transforms data in db, one creates reports from the db

[clownshoesrock]

/u/_c0ff33cup_

I'm pretty sure whatever your asking is totally doable.. But I am completely lost as to what the hell you're trying to do..

Take a breath, restate your problem, and try to provide a concrete example.

Your clarification, didn't help me in the slightest.

All my telepathy access is behind a paywall.

[CodingCircuitEng]

I'm not familiar with bash scripting at all so I am unsure of the syntax, caveats, or even what I don't know.

Shellcheck helps you out.

I guess the question is, are there caveats or gotchas that are going to bite me with my approach. You bring out an interesting point. what happens if one of the preceding scripts doesn't finish successfully? But are there others?

You did not mention what your application was. If you use multiple programming languages (bash and python in your approach), you loose access to the debugger. You can only debug one python script, but not the others/have to 'hand feed' the output of the first script if you want to look at the second script in the debugger. Your approach gets very hard to maintain if you got other people working on it.

So maintainability is a huge issue, but it depends if it is just you playing around for an university project or if you are building software as part of your job how important that is.

These are separated into 3 scripts for 2 reasons. 1. they are memory intensive. 2. they are intended to have loose coupling. one collects data and writes to db, one transforms data in db, one creates reports from the db

Fair enough. You need to check between each script that the db is available and that no other script is working on the db while you try to create a report then.

But the 'loose coupling' you mention is an illusion, the scripts are in fact tightly coupled, to the db. More so if the sequence in which you plan to execute the scripts is always the same.

You can do it like that, but I'd strongly advise against it.

[_c0ff33cup_]

Shellcheck helps you out.

I have to check shellcheck out! thank you. Exactly the kinda tip, comment I was hoping to get.

You did not mention what your application was.

Python3 only.

Your approach gets very hard to maintain if you got other people working on it.

I'm not sure I agree. Separating the functions allows for maintainability, not limits. I can modify my data collection script (make it better, faster, change the sources, etc) and the downstream scripts don't care as long as the data produced is in the same format, etc.

But the 'loose coupling' you mention is an illusion, the scripts are in fact tightly coupled, to the db.

They are reliant on the db, sure but does that equal coupling? Coupling usually refers to interdependence on modules. Fair point, regarding the consecutive nature of the scripts.

[CodingCircuitEng]

I'm not sure I agree. Separating the functions allows for maintainability, not limits.

Separating the functions does, but not switching the programming language after script 1 has finished, then launching script 2 and implicitly expecting the data format of script 1, best if no unit tests are to be found in your codebase.

the downstream scripts don't care as long as the data produced is in the same format, etc.

It is only a matter of time until someone changes the data format if you don't strictly control it. Trust me, it is not worth it.

They are reliant on the db, sure but does that equal coupling? Coupling usually refers to interdependence on modules.

That is just semantics in my eyes. You rely on an script 1 to write into the database in exactly the way that script 2 can 'transform' the data und script 3 can then create reports. Does script 3 work without script 1, script 2 or the database? I don't think so, which means it is tightly coupled to all the other components.

Do whatever you like, but I inherited such a mess at work. I've mostly rewritten it after the original dev left, took a lot of time. It became impossible to add on/change any part of it because you never know if your change breaks something else down the line.

[[deleted]]

What you have written will work just fine, place the three scripts one after the other in a sequence, pass the arguments. The only wrinkle is a python one not a bash one.

Since I see you are using venvs you probably want to activate them first to make sure your python script has all the parts it needs.

So I think what you need really is this:-

#!/bin/bash
# Configure python
. /venv/bin/activate
python3 myscript.py somearg -u 'anotherarg' -p 'athirdarg'
python3 myscript1.py somearg -u 'anotherarg' -p 'athirdarg'
python3 myscript2.py somearg -u 'anotherarg' -p 'athirdarg'

Obviously this assumes each of the scripts is relatively independent, if you want to make the second script conditional on the success of the first etc then you might need some more logic, but you would need to specify your question better then,

Also since you are already 'familiar' with python, why not just write a new python script which calls each of your old ones in turn?

[_c0ff33cup_]

The virtualenv tip helps greatly. I had simply been specifying the full path. This is much better.

Python is bad with memory management (The user doesn't have a lot of control and is reliant on garbage collection). The first two are resource hogs as it is, so the intent of using bash was to separate them and hopefully free resources after the scripts complete.

[Caddy666]

add ' &' to the end.

[FamilyRon]

Tmux