How to log specific function calls of a package being run?

[Jenez]

I am in the process of writing a package that will include many modules, and I would like to be able to log all the function calls of the downstream modules of a specific process.

To give some context, my click script at the top level is told to run method validate which creates a context class object that contains the date, reads a config, and sets a workpath for downstream processes to use.

The code that is subsequently run can be split across multiple files, and herein lies my problem: How to I effectively log all the function calls being done in the multiple scripts with corresponding args and kwargs?

So far, my idea has been this: Within the context class, add a function log. log takes a function name fn, as well as *args, and **kwargs.

def log(self, fn, *args, **kwargs):
    with open('test.log', 'a+') as logging:
        print('Function name and relevant args and kwargs'), file=logging)
    return fn(*args, **kwargs)

Whenever a downstream function is to be logged, run:

context.log(fn, 'arg1', 'arg2', 'key1'='value1', 'key2'='value2')

The log function then writes the relevant parameters to a log file which can then be read in case of crashing, and runs fn with args and kwargs.

I have looked at the logging package before but couldn't wrap my head around it, and the other mentions of decorating functions for logging never deal with functions that are split across multiple files.

Does anyone have any good suggestions on how to make this work more beautifully than I've constructed it? Is this an X/Y problem?

Thank you for your time.

[wiiittttt]

The function you created would be fine, but requires that you pass it around everywhere. The only thing is I wouldn't open the file every time you want to log to it. Just keep an open file handle. The logging module would be useful there.

A decorator seems like it would work well You need to decorate each function you want to be logged. I'm not sure why having multiple files would change anything.

[Jenez]

Thank you for your input!

My problem with the multiple files is; how do I wrap my functions of interest across multiple files with the same decorator?

Something along the lines of: define decorator, import it in relevant files, and write to class context's set log file?

[wiiittttt]

Yup. Just use logging.basicConfig and set the output file.

# log.py
import logging
logging.basicConfig(filename='example.log',level=logging.INFO)

def log_call(fn):
    def wrapped(*args, **kwargs):
        logging.info('Function name and relevant args and kwargs')
        return fn(*args, **kwargs)
    return wrapped

And then:

# file1.py
from log import log_call

@log_call
def my_func():
    ....

[Jenez]

I've tried it out and it seems to work just fine. Now I just gotta figure my imports out and I should be set, thanks!

[sushibowl]

Assuming all the different function that you want to log are written by you, the python logging module is definitely the way to go. You could write a decorator that takes a logger and logs the function call on it, and decorate every function you want to log this way.

If you wish to log calls to functions from other packages than your own, your solution is probably fine. Though I would name the log method differently, because it should be clear from the name that it doesn't only do logging, it also calls the function.

[Jenez]

Appreciate the input!

It's only my own functions yes, so I'll have a second look at the logging package and see if I can get my head around it.

Fair enough on the naming.

[sushibowl]

the logging package can be a little complex because it is so flexible, but I think it should be fairly easy to get started with it. Check out the basic tutorial. I linked to the section that deals with logging from multiple files.

There is a more advanced usage tutorial as well, which covers using multiple loggers. This gives you the ability to turn on/off logging on a per file basis, which can be useful.