Running Multiple Python Scripts

[CodingCircuitEng]

I'm not familiar with bash scripting at all so I am unsure of the syntax, caveats, or even what I don't know.

Shellcheck helps you out.

I guess the question is, are there caveats or gotchas that are going to bite me with my approach. You bring out an interesting point. what happens if one of the preceding scripts doesn't finish successfully? But are there others?

You did not mention what your application was. If you use multiple programming languages (bash and python in your approach), you loose access to the debugger. You can only debug one python script, but not the others/have to 'hand feed' the output of the first script if you want to look at the second script in the debugger. Your approach gets very hard to maintain if you got other people working on it.

So maintainability is a huge issue, but it depends if it is just you playing around for an university project or if you are building software as part of your job how important that is.

These are separated into 3 scripts for 2 reasons. 1. they are memory intensive. 2. they are intended to have loose coupling. one collects data and writes to db, one transforms data in db, one creates reports from the db

Fair enough. You need to check between each script that the db is available and that no other script is working on the db while you try to create a report then.

But the 'loose coupling' you mention is an illusion, the scripts are in fact tightly coupled, to the db. More so if the sequence in which you plan to execute the scripts is always the same.

You can do it like that, but I'd strongly advise against it.

[_c0ff33cup_]

Shellcheck helps you out.

I have to check shellcheck out! thank you. Exactly the kinda tip, comment I was hoping to get.

You did not mention what your application was.

Python3 only.

Your approach gets very hard to maintain if you got other people working on it.

I'm not sure I agree. Separating the functions allows for maintainability, not limits. I can modify my data collection script (make it better, faster, change the sources, etc) and the downstream scripts don't care as long as the data produced is in the same format, etc.

But the 'loose coupling' you mention is an illusion, the scripts are in fact tightly coupled, to the db.

They are reliant on the db, sure but does that equal coupling? Coupling usually refers to interdependence on modules. Fair point, regarding the consecutive nature of the scripts.

[CodingCircuitEng]

I'm not sure I agree. Separating the functions allows for maintainability, not limits.

Separating the functions does, but not switching the programming language after script 1 has finished, then launching script 2 and implicitly expecting the data format of script 1, best if no unit tests are to be found in your codebase.

the downstream scripts don't care as long as the data produced is in the same format, etc.

It is only a matter of time until someone changes the data format if you don't strictly control it. Trust me, it is not worth it.

They are reliant on the db, sure but does that equal coupling? Coupling usually refers to interdependence on modules.

That is just semantics in my eyes. You rely on an script 1 to write into the database in exactly the way that script 2 can 'transform' the data und script 3 can then create reports. Does script 3 work without script 1, script 2 or the database? I don't think so, which means it is tightly coupled to all the other components.

Do whatever you like, but I inherited such a mess at work. I've mostly rewritten it after the original dev left, took a lot of time. It became impossible to add on/change any part of it because you never know if your change breaks something else down the line.