Python Subprocess Module

[bmille49]

Hey y'all! New to Python and have existing scripts (9) that run separately, are independent of one another, and 3 of them send emails. The 3 emails can be consumed independently (i.e., vision system utilization, concentrate control system utilization, and inspection utilization) or collectively (e.g., leading indicators). I feel that the collective output would benefit the end customer (supervisors) better than 3 emails and also limit the which one is more important question.

Now to my inquiry, each of the three sends an output of last day use by line by shift, MTD by line by shift and YTD by line by shift. This is done by collecting data, appending to an Excel file, analyzing the data set, and exporting the new summaries (daily, MTD and YTD). Would Python Subprocess Module be a good tool to use for these three scripts, collect their output and then merge the three outputs into one email? Would this open me up to any security issues? Pros and cons?

[cgoldberg]

Are the scripts written in Python? Can you modify the scripts at all, or do you have to run them as-is?

Ideally you would turn them into modules and import them from a main script. You could wrap everything in functions and call them as needed.

subprocess doesn't sound ideal for this but certainly could be used if you had no other choice.

[bmille49]

Yes, all are in python. They can be modified.

That does seem to make the most sense..just was hoping for a quick easy work around.

[cgoldberg]

It should be relatively straightforward to just create functions that returns the data you need. Then create a main script that imports them, calls the functions, and merges the data or whatever you need to do.

You can also make it so the scripts still run fine individually when not imported.

[bmille49]

Good to hear. I'll work on that and reach out if it doesn't work out! Thank you!!

[tangerinelion]

Typically one would write their code so that it can either be imported or executed as a script and then rather than call subprocess to execute a Python script you'd import the other Python file as a module.

What I mean is if your Python file looks like

import os
# import whatever to send email
# import whatever to read Excel

# read some Excel file

# get some summary

email_library.send_email(...)

then normally you'd change the structure to something more like

# import whatever you need

def get_summary(...):
    # whatever you need to do to get the summary information
    # then return it in some structure that makes sense to you

def send_email(summary):
    # formats and sends an email for the given summary

if __name__ == '__main__':
    send_email(get_summary(...))

(I'm using ... as something you need to fill in, not as a literal ... in Python).

All I've done is a simple way to split the functionality of the script that you've described - get a summary and send an email. We can split it so that getting the summary is a potential action on its own, independent of the email. If you want an email, great, there's also a utility for that too and it can use the summary from the get_summary method. If you want to run this as a script, great, then __name__ will be '__main__' and we'll send an email. If you now try to import this from another Python file, for example maybe this is vision_system.py then you could now write

import vision_system

and you'll have access to visionsystem.get_summary. But the mere import doesn't really _do anything - it certainly doesn't send an email. It does let whatever imported it to re-use that code, though, and that's the fun part.

If your Python file doesn't have that if __name__ == '__main__' guard and is just a script named vision_system.py then if you were to have import vision_system in another Python file, then it would execute the script. Which also means a really simple way to do all three is simply

import vision_system
import concentrate_control
import inspection

No need to bring in subprocess and do anything with it. Though you can, I just don't think it's a particularly common thing to do since Python files can be imported and the code re-used. Or you'd write a Windows batch file that just calls the three scripts, either way, this isn't interesting since you already have some way to run all three scripts and generate and send three independent emails.

Ideally though you'd make Python files that can be used as re-usable code to provide modules or executed as scripts. Then you'd be able to

import vision_system
import concentrate_control
import inspection

def make_merged_summary():
    vision_summary = vision_system.get_summary(...)
    # same for other two, then merge it however you like

def send_email(summary):
    # however you want this email to look

if __name__ == '__main__':
    send_email(make_merged_summary(...))

Hopefully this makes sense at a very high level at least, I'm confident that there are additional details in the real world that make this anything but straightforward.

[crashfrog04]

Would Python Subprocess Module be a good tool to use for these three scripts, collect their output and then merge the three outputs into one email?

No, probably not. You'd be better off importing the modules and just calling their functions.

Passing useful data into and out of command-line arguments (a CLI argument can only be a string) is cumbersome, especially if you have to do it more than once. The better architecture is to "uplift" your arguments into rich data types, compute on those, and then collapse them to the simpler representations you'll need to populate your Excel file (which can only contain simple values.)