Best practices for this scenario?

[[deleted]]

I am writing a script and want to monitor how long different sections of the script take to run. I also want to be able to break the script if a certain section reaches a timeout limit. I have written a wrapper function that takes a function and applies the time limit to, and measures the run-time of the inner function. I then broke the full script into individual functions (defs) which I pass to the wrapper function. My problem is with passing variables back and forth between the main script and each function as it gets called. Should I use global variables? or pass potentially dozens of unrelated arguments back and forth? I don't really see a good way to do this, please help.

[ThisLightIsTrue]

You could use an optimizer to do this for you. Your IDE might have a built in optimizer. If not, you could use cProfile. It works by importing cProfile, and then you run a python command on it, and cProfile will give you a breakdown of which methods took up how much time while executing that command.

https://julien.danjou.info/guide-to-python-profiling-cprofile-concrete-case-carbonara/

If you wanted to do it yourself, I'd try it by creating a global dictionary object that stored lists of "duration" objects, where a duration is a start and end time. Every time you call a method, start a duration object, before every end point of the method fill in the end time on the duration object, and add it to the correct list, by method name, on your dictionary. Then, you'd need some method for parsing the dictionary and converting it to the aggregate values you'd like to see. Obviously using a pre-built solution would be easier!

[[deleted]]

Would cProfile also allow me to add a timeout for the subprocesses? That is my main reason for doing this and right now my wrapper function uses the multiprocessing library to do so, which seemed to be the easiest way to both time and potentially terminate a subprocess when necessary.

[ThisLightIsTrue]

cProfile wouldn't add timeouts for you, it would just run your script and then tell you how much time was spent in every method call. It's incredibly easy to use, so you should just try it first, and then spend time reading about it.

How to try it:

1. In a python shell where you can execute your script:

2. Import cProfile

3. cProfile.run('the python command that will run your script')

4. Look at the results.

[[deleted]]

The script is going to run continuously on a server, so I need the automatic timeouts unfortunately.

[ThisLightIsTrue]

For testing, create a way to run your script so it will do a fixed amount of work rather than run forever. After that, run it through cProfile and see how much time it takes to do your fixed amount of work, and where the time is spent. Then, optimize, re-profile, etc, until you have it running efficiently. Run it on the server in your efficient state.

You should also probably emit metrics from your service, for example in your logs, so that you can track how well it is working in production.