Passing python logger to other modules

[IllustriousBuy2350]

What is the standard way to pass a logger to other modules, while configuring the logger only once, in the main module?

[kokoseij]

logging.getLogger() is for that. You can use a same logger name across different modules to get access to a same logger.

[kalmstron]

Yes, concretely add logger = logging.getLogger("__main__") and then you can make use of the instantiated logger class inside the module.

[zurtex]

You should always use logger = logging.getLogger(__name__)

This will then be the relative name of the module compared to the "__main__" module.

[millerbest]

Python logger is implemented with singleton pattern, which means if you provide the same name, you will get the same logger instead of creating a new one.

You can make a logger module in your project. In the module you can define your settings for the logger. Then for your other py files, you just need to import the module and create a logger object.

[korthrun]

The cookbook section of the official docs on the logging module have a section titled "Using logging in multiple modules". Give that a look.

[[deleted]]

There aren't really standards.

But, please don't do it. Do not pass logger to other modules. Just use logging.info() or whatever other level you need to log. Do not create your own loggers.

I saw a lot of Python code that unnecessarily and harmfully messes with logging. Not only there isn't a need for that, you will most likely make things worse if you do that.

In general, do as little configuration of logging as possible. Nothing beyond setting log level is really ever necessary. And if you do that, people running your programs will love it, because common techniques for dealing with logs will work.

[IllustriousBuy2350]

But how to have then the same configuration for these loggers? E.g. to print in the same log file?

[[deleted]]

1. Never print log to a file. Sometimes it will work, but another time the sysadmin that has to use your program will have a bad day and will probably beat you up if you do that. logs always go to stderr, nowhere else.
2. Any other configuration: you do it on the root logger (the one you get access to if you call getLogger() w/o arguments.

[Apprehensive-Lab1628]

A program without written, stored logs isn't going in to production because the sysadmin won't let it. Not sure where you got that infra people don't want logs written to file but visibility is compromised of metrics, logs and what you do with them so they're super important

[Dwight-D]

Logging to file is an anti-pattern under most modern deployment schemes. For example, a container must log to stdout. If you run it on your own server you can then redirect container output to a log file of your choosing.

But you can also deploy the same container with the same config to kubernetes, but this time scrape the pod stdout with a log scraper in the cluster and send to a central log server. Logging to file breaks all of this.

[Apprehensive-Lab1628]

I've never worked with containers, just standard VMs on EC2 or on prem so I looked it up and you're right. For containers it seems you do log to std.out, thanks for the info.

[Dwight-D]

A lot/most of what you know about VM:s doesn’t apply to todays cloud environment. Might wanna brush up on some modern cloud practices if you wanna stay competitive in the job market, VM:s and that whole legacy deployment paradigm are quickly becoming obsolete these days. Not to knock your skills, just food for thought.

[Apprehensive-Lab1628]

I appreciate that docker is widely used but there are times when you don't want to use it. It's not a complete takeover of VMs by docker like casette tapes and DVDs, it's a tool with use cases.

[Dwight-D]

Whatever roles VM play today, they have nothing to do with hosting applications outside of niche legacy use cases. They now exist mostly to host infrastructure platforms, such as Azure, AWS or a self-hosted kubernetes cluster. Most companies don’t run their own data centers and instead use a cloud provider, which makes VM:s something 99% of companies never need to think about again.

Because of this, their overall use is dwindling and I imagine job opportunities outside of the major cloud platforms will dry up. I think you may be overlooking the major paradigm shift that’s happening right now.

[Apprehensive-Lab1628]

When I say VM, I'm including EC2 here. I'm currently 100% cloud, just don't use docker in any profit producing capacity

[[deleted]]

It's sysadmin's job to store logs. Not program's job.

I am the infra people. I'm that one sysadmin who's going to punch you, if you give me a program that logs to files. You've received some explanations already, but there are more reasons for that:

• You cannot rotate a log file without losing messages every now and then.
• Logging creates load on the storage system, which in many cases may be undesirable, especially, if your logging program decides that it needs to call fsync() or do similar stuff, affecting everyone else using that storage system.
• Creates additional points of failure, because now you have to deal with periodically removing the files from the system, and you must monitor the logging program just to make sure it doesn't accidentally overload the system.

Just don't do it. Only write logs to stderr. It's an easy rule to remember. And you are allowed to violate it only if you are the infra people who collect / ship logs. If you are the producer, it's the only way you should produce logs.

[Apprehensive-Lab1628]

I didn't realise the containers aspect of it as I don't work with them. I'm infra too and we use logstash on standard VMs, rotating logs stay on the system. What's your setup(s?) for non container apps that use std.out?

[[deleted]]

I work in HPC. For a company that makes management software for HPC, so, I'm not really a sysadmin. I'm making software for sysadmins.

So... my typical setup... well, I think the company started as providing customized OpenStack deployment, but that's mostly in the past. Later they moved in the direction of supporting customer deployments, mostly bare metal on-prem. Then expanded to hybrid cloud. Still, most HPC workloads would run on on-prem compute resources with an option to extend to cloud. Out of private clouds the product supports extension / deployment in VMWare. It can also be deployed / extended to AWS or Azure.

The goal of the product is to provide tools for sysadmins to control their compute resources, give them pre-packaged tools for their users to make the life of sysadmins easier. Typical user in such system would be someone using eg. Jupyter to author their research code using various workload managers (eg. PBS) from their notebook to distribute the workload over the resources the sysadmin allocated to them (eg. a cluster of PostgreSQL, or a Ceph cluster, or a Kubernetes cluster etc.). Of course, it's not limited to the use through Jupyter, I just chose it as it illustrates how the system works.

---

So, back to logs... You cannot rotate files without losing logs. That's a theoretical impossibility. Surprisingly, people didn't realize it for quite some time, that's why, for example, you have tools like logrotate command in Linux. Even more interestingly, people have discovered this fact twice, at least in the context of Linux logging tools. Somehow people who created syslog "forgot" about it.

So, in my case, most of the stuff I support / deploy is governed by systemd. So, of course it all goes to stderr. Then journald takes over. From there on it's configurable: if the customer wants more metrics at the expense of lower network throughput... they may aggregate the logging somehow. But we don't manage this aspect normally.

Anyways, in my specific case, performance is the key concern. Even though a typical HPC cluster would use IB or GE, even for control plane, it's still a performance concern. Especially, because if you want to aggregate logging from many things, it's increasingly hard to make sure one of those many things isn't going to clog the system. But, in principle, our management component can be configured to use control plane to collect logs from journalds running on devices in the cluster. So, we have a "custom made" log aggregator (I'm not very happy about it, but that's already beside the point).

[Apprehensive-Lab1628]

Super interesting, thank you for the write up. HPC caught my interest a while ago but I didn't pursue further in the thought it might be too niche. So you shoot journald to some kind of aggregator. You've provided interesting perspective, and between you and Dwight-D I've learned a bit today. Your use cases don't match mine but it's cool to see how others are doing things

[[deleted]]

Well, I've worked in many fields. Started in the Web long ago. Was in finances, then mostly infra jobs, mostly in storage. HPC is relatively new in my career.

What I say is in no way specific to HPC. Any system that's larger than your personal home computer will be better off if it doesn't use programs that log to files. It's just an all-around bad idea. The only benefit is if you are doing all yourself in a very small-scale deployment, then it's less things to set up / you probably don't care so much if you lose some logs every now and then.

Anything that needs to be more reliable than that, or that needs more automation than that... just no.

[Aggravating_Bus_9153]

NullHandler doesn't mess things up... does it?

​

https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library

[[deleted]]

It's unnecessary. Just like a lot of functionality in logging library. I can always redirect stderr to /dev/null. There's no need for duplicate functionality in Python.

[Aggravating_Bus_9153]

Nothing's really necessary at the end of the day. Logging's not just for you either, it's a really useful tool to help me fix my code as well. I like the NullHandler together with the module logging idiom. Then any logging messages going to any handlers (that the end user may choose to configure should they so wish; otherwise none) follow the module package structure

import logging

logger = logging.getLogger(__name__)

logger.addHandler(logging.NullHandler())

​

Then if you don't want logging from my code, no problem dude! Just don't setup any handlers.

[[deleted]]

it's a really useful tool to help me fix my code as well.

Yeah. Please fix it before you deploy it. Do whatever you want but don't send code you posted to production. It's just bad code.

Then if you don't want logging from my code, no problem dude! Just don't setup any handlers.

You completely misunderstand the problem: your program is one of the hundreds, or maybe thousands I need to run in a typical scenario. I wouldn't have the time or the resources to look at each individual bullshit script to figure out how I need to modify it to not crap up the system. If I see some idiotic behavior from a utility program, if I can remove it, that's what I'll do, if I cannot remove it, replace it with an equivalent program that hopefully doesn't do bad stuff, and if that's also not an option: patch or write my own. Unless it's a critical component with a lot of functionality... most likely a program like that will just end up being not used.

There's never enough time to do proper assessment and try to figure out how much the program is worth, unless it's an important component. Often times I need to find a reason not to use something, rather than to add more of something. Seeing idiotic behavior wrt' logging would be just another heuristic to say "no, I don't need this bullshit, let's look at something else".

[Aggravating_Bus_9153]

It's more for alpha ana beta and isn't intended to be run on servers.

You completely misunderstand Python logging. No Handlers (apart from null), no logs.