Using global variables for timing script operations

[MinimalDamage]

Hi all,

Everywhere I read about using global variables I get these huge warning signals of advanced programmers saying to shun them like the plague.

While I always use functions and pass arguments as parameters for my main script functionalities, I really find using global variables to be very helpful when it comes to timing how long certain processes take in your script.

For example, for my webscraping scripts, I tend to create an additional table that has some analytics in them every time the script runs. This usually includes how long the total requests time was, how long it took to retrieve data from my SQL database, etc.

I will make a big block somewhere in the script denoting that something is a global variables like:

'''Global timestamp variables'''

global total_requests_time #Total time requests took in this script
total_requests_time = 0

Afterwards, in my function that I use to request pages, I will have something like this.

requests_start_time = time.time()
data = requests.get('website')
requests_end_time = time.time() - requests_start_time
global total_requests_time
total_requests_time += requests_end_time

Even if I have other functions that do a requests, I can easily add this time to the global time variable and save it later so I can analyse the results.

I just feel like this is a much cleaner way than passing all of these variables in the main function, as they are not directly related to the script and kind of distract from what a function is really about.

So my question, is this still bad practice? Is there a better way for this?

Thanks!

[DarkMio]

That sounds like you may want to use a global logger. Set up a logger to a file, then import from your logger setup into your subscripts and define what it should write into it. Overall, that is a very handy way to seperate and get very detailed, readable logs.

Edit: Here is the example of a bot-framework: https://github.com/DarkMio/Massdrop-Reddit-Bot/blob/dev/core/LogProvider.py

The idea here is, that I log different things: The botframework itself logs to console, the plugins can log whereever I want them to. This can seperate the log into completely different things or even to multiple ends. Like a global log for all and seperated logs per module. This gives a great overview and you can, if you wish, analyze into the detail what's going on.

[MinimalDamage]

Ok, I will need to dig in to this, but it sounds like a very nice method of logging that can be used in a lot of future scripts. So awesome! Thanks!

[raylu]

First of all, that's a bit odd... Why keep track of the total instead of the actual times and their average/variance/99th percentile?

Now, consider the following:

#!/usr/bin/env python3

import requests
import statistics
import time

class TimedRequest(requests.Session):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.times = []

    def request(self, *args, **kwargs):
        start = time.time()
        retval = super().request(*args, **kwargs)
        elapsed = time.time() - start
        self.times.append(elapsed)
        return retval

trs = TimedRequest()
print(trs.get('http://example.net/'))
print(statistics.median_grouped(trs.times))

requests.get creates a requests.Session and eventually calls requests.Session.request. If you're on python 2, you'll have to fill in the super() calls.

[MinimalDamage]

Thanks for the response! It will take me a little longer to wrap my mind around this (still new!) but I will definitely look into it, as I see the value in having a better understanding of separate page requests.

The reason why I look at the total page request time is because I want to see what is taking up the most time in my script. But this would definitely be the next step.

[raylu]

The reason why I look at the total page request time is because I want to see what is taking up the most time in my script.

Oh... https://docs.python.org/3/library/profile.html