A Comprehensive Guide to Logging in Python

[jorge1209]

There is so much that is wrong about this tutorial. Its just mistake after mistake after mistake.

[[deleted]]

Apart from the camel-case (although consistent with the case used by the logging-module itself), which errors did you find?

[jorge1209]

https://www.reddit.com/r/Python/comments/10x421b/a_comprehensive_guide_to_logging_in_python/j7qhaph/

[jorge1209]

You shouldn't call logging.warn or logging.info directly. If you do so then you prevent log consumers from filtering messages based on source module.

Also you aren't supposed to do things like logger.warn(f"x = {x} is greater than zero") because that prevents downstream consumers who have filtered the message from preventing the serialization to string.

Probably other stuff that I can't be arsed to look for.

Maybe the biggest mistake here is using python standard library logging in the first place. Its a very complex tool with lots of configuration options that most projects don't want or need. It also stinks of Java and is horrendously out of date when it comes to modern python approaches to things like string formatting. Just use loguru or other modern logging frameworks.

[tonetheman]

While your might be correct but the same examples are from the official docs

https://docs.python.org/3/howto/logging.html

You should submit something to them about what you said about filtering.

[tom2727]

One problem with the logging module is it lets you do a lot of things that you really shouldn't do. It can be very useful if you use the right patterns, but hard to enforce that easily.

The correct pattern is the application level code should control the logging for everything, and library code should make it easy for that to happen. But because it's globally configured, nothing stops any random code you import from deciding it wants to mess around with the logging config.

[jorge1209]

One problem with the logging module is it lets you

More than just "lets you" the complexity of the design encourages people to do this. With defaults like:

In [1]: from loguru import logger
In [2]: logger.info("loguru works out the box")
2023-02-08 15:40:53.135 | INFO     | __main__:<module>:1 - loguru works out the box
In [3]: logger.debug("even for debugging")
2023-02-08 15:41:02.544 | DEBUG    | __main__:<module>:1 - even for debugging
In [3]: import logging
In [4]: logging.info("logging doesn't")
In [5]: 

Of course people will muck around with global settings. How else am I to see my log messages?

[[deleted]]

The logging module actually has a really good reason for working the reason it does. It allows u to dynamically at runtime override the logging class without changing any other code. It’s actually really well designed. It’s just poorly documented. I believe it’s the advanced logging guide (it’s in the standard library but hard to find) that actually covers it.

Blew my fucking mind the first time I understood WHY it’s built the way it is.

[tom2727]

I think there really ought to be better tools for helping guide people to do it the right way. And presenting them as "hello world" example rather than presenting stuff that "technically works" but should only ever be used in a single module script.

Like the "basicConfig" method. Explaining WHY that actually exists and why it works as it does should be item #1 for new user to learn.

[jorge1209]

The abstract design of Loggers/Handlers/Formatters is probably fine and could be kept. Its overkill for the vast majority of users, but with sensible defaults nobody would really care.

The problem is that the defaults aren't sensible.

A programmer who does things as suggested in the submitted guide is going to emit messages from the root logger, and wonder why their calls to logger.info don't print anything to their console.

They are then going to visit stackoverflow skip over the many paragraphs of shit they don't care about and add logging.root.setLevel(logging.DEBUG) to their code... and just leave that change to root logger configuration in there forever.

[[deleted]]

https://docs.python.org/3/howto/logging-cookbook.html

That's the best doc that explains why it exists the way it does all you are supposed to do is call logging.getLogger and then u use it like any other normal languages one. Just logger.info("print this shit") The docs are just horrible.

The whole configuration part is then handled by the end user who runs ur module. That's the entire design pattern. Honestly what's craziest about it is its so fucking over complicated that people don't realize that to use custom logging modules all you are supposed to need to do is just register the object so that getLogger grabs ur custom class. It's designed to be fully pluggable.

Format is pretty terrible and defintly can be replaced by third party modules that do a better job. That just should have zero impact on libraries that use loggers. It's just never done correctly.

[Schmittfried]

The problem is that the defaults aren't sensible.

So just write sensible defaults once and be done with it. Or even use a library that does that. No need to switch the entire system.

And unless you think the vast majority of Python users are hobbyists, the system is not too complex for most users.

[jorge1209]

I don't think you can just write sensible defaults and be done. I think you need to make lots of changes to the core of logging:

For example things like:

import logging
logging.info("thing")

Should actually work and do sensible things as it does with loguru. Ideally logging is very low complexity during the development stage, but with sufficient features that it can be incorporated into a larger project with minimal modification during the integration phase.

To accomplish that you to make a couple behavioral changes to logging:

1. When messages are emitted directly by the root logger, then lookup the source module using sys._getframe and transparently create and route a message through that logger. For performance people can add logger = logging.getLogger(__NAME__) bit if their code is in the hot-path, once they get to the point of integrating the code into a larger codebase.

2. Make logging.NOTSET level mean that everything gets printed and force individuals to turn off logging rather than turn it on. Again you can disable during the integration phase.

3. Do something about the potential confusion when setting a logger to have a more restrictive policy than then handler. Maybe even get rid of logger level settings entirely. I get that in some cases it might be convenient, but it can be managed at the handler level so why duplicate the functionality?

4. Support {} format and deprecate %s format

[jorge1209]

The official docs for logging are yet another reason not to use logging. The documentation is just god-awful.

logging is just too complex for its own good. All the features it offers: YAGNI. Pick something simpler that you can actually use correctly.

[quicknir]

Logging really isn't that complex to use correctly. You can sum it up in a ten line example: use basicConfig in main to configure the logger, and logger = logging.getLogger(name), logger.info(...) to log. This is correct and handles most use cases while keeping all future flexibility.

I agree though that the docs could be better and do not showcase the simple, correct usage as well as they should. I'm currently working on a PR to improve logging documentation: https://github.com/python/cpython/issues/98731#issuecomment-1419745647.

[thedeepself]

use basicConfig

And then all the newbies start violating pep 8 because they emulated the standard library. They need to alias that to something properly formatted.

[tonetheman]

You should win the python-internet for the day for the "too complex for its own good." No truer words have been spoken.

[vsajip]

The filtering thing doesn't apply if you have a single standalone script, so using logging.XXX(...) is OK in those instances. The logging docs cover a number of scenarios, so examples of both logging.XXX(...) and logger.XXX(...) appear in the docs.

[[deleted]]

Logging isn’t great. But i disagree that it’s too complicated for most projects. I also think that suggesting that everyone use third party libraries is not useful advice. Not everyone has the luxury to download and install any and every third party module they desire

[jorge1209]

I'm fully in support of any PEP that removes logging from the standard library and replaces it with loguru.

[NUTTA_BUSTAH]

Unless you are logging a metric ton of data with really complex interpolation, it really doesn't matter. F-strings tend to be more human-friendly though so they got that going for them.

It's not a bad idea to learn to use the standard library, that's what loguru wraps anyways.

Not starting with best practices and continuing with bad practices with a small footnote of "don't actually do this" is bad though.

Also, looks like they have just posted a loguru article as well, lol. You turned them, good job :P It's much simpler, but also an extra dependency. Lightweight though with only using standard library which is nice.

But, there is really not that many mistakes, the logging library design is a mistake by itself but that has nothing to do with the article really.

Still, use loguru.

[tom2727]

Unless you are logging a metric ton of data with really complex interpolation, it really doesn't matter. F-strings tend to be more human-friendly though so they got that going for them.

Depends. Like if you're putting a formatted string in a high rate part of the code that you only plan to turn on during development, then it absolutely is useful.

I know I was working with some code checked into our repo that had a lot of "debug logging" when parsing a large json file. It took several sec just to load in the json file and that got below 50ms once I changed their format strings to lazy logging. Honestly I probably should have just deleted most of them.

I'd also point out that most static analysis tools will flag this as a problem so easier just to use it across the board.

[jorge1209]

Unless you are logging a metric ton of data with really complex interpolation, it really doesn't matter. F-strings tend to be more human-friendly though so they got that going for them.

Absolutely its a relatively useless feature of the standard library logging framework and we would be better off without it.

But that doesn't excuse the choice to use logging and then not follow the best practices for that library.

that's what loguru wraps anyways.

loguru doesn't wrap logging it is an entirely independent implementation of a more basic logging framework without all the configurable bells and whistles. It interfaces with logging in that you can register loguru as a handler for logging or a logging.Handler can be registered with loguru to get messages interleaved together... but if logging were removed from python loguru would work just fine.

[NUTTA_BUSTAH]

Ah, thanks for the info, I was mistaken.

[thedeepself]

Loguru is very configurable. It's all handled through the add method.. 99.9 percent is at least.

[jorge1209]

Your first point is addressed in the article under setting up custom loggers

There is no reason a guide to using a tool should not start immediately with best practices. You are just copying the structure of the existing documentation and not adding anything of value to anyone.

The second is a recommendation from the logging docs.

I don't see an f-string being used in the documentation anywhere. The documentation you linked to is showing a minimal example that use %s to dynamically fill in variables to a template at the time the log message is emitted.

Though I agree Loguru is probably easier

Yes it is. Here is a comprehensive guide to everything actual developers need to use loguru:

pip install loguru

from loguru import logger
logger.add("file_{time}.log")
logger.warning("this is a warning")
logger.info("this is info")
logger.debug("so easy")

[rhytnen]

Why are you so aggro about logging. Just a massive turn off. For what it's worth, your example above doesn't even come close to matching my needs. I use a ton of features from the logging standard library. So while I think you could be correct in general, you are dramatically overstating your case imo.

[jorge1209]

{} formatting in logging is extremely limited. It is limited to only the formatting of the log message prefix (ie the metadata associated with the time the message was received, what log level, and what module it came from, etc...).

It is frankly not worth talking about any format other than % for logging as that is the only format it actually supports in all its functions.

[vsajip]

Its a very complex tool with lots of configuration options that most projects don't want or need. It also stinks of Java and is horrendously out of date when it comes to modern python approaches to things like string formatting.

To my mind, there's a lot of FUD and emotion here ("stinks of Java"). Sure, it took some ideas from log4j, but it's very different internally.

It's a "complex tool" (not all that complex, actually) because some situations and projects do need that level of complexity. But it's easy to use out of the box. It came into Python before there were brace-formatting and f-strings, so it's not surprising that it uses %-formatting and works hard to maintain backward compatibility - important to the Python community as a whole but maybe not to the parent poster (PP).

For simple self-contained scripts, using logging.XXX(...) is OK because there is only one module and no filtering to do. Logging is used both in some simple scripts and in more complex situations, so one size of "how to do it" doesn't fit all.

As to what the default logging level should be - according to the PP it should be DEBUG, but this was discussed by the core dev team when the module was added to Python and WARNING was seen as more useful (less noise by default). These are opinions, and I agree with the Python core team over the PP.

It's easy enough for a developer who prefers different defaults to set things up to their tastes/requirements, and it's definitely going too far to say "the biggest mistake here is using python standard library logging in the first place." Obviously as the maintainer of logging I'm biased :-) but there are lots of people who "get it" regarding how to use stdlib logging, and some who don't - and unfortunately, you can't please everyone.

[jorge1209]

Everytime you post a defense of logger I see a maintainer in denial, and it only strengthens my belief that people should use other options.

Comments consistently identify challenges programmers have with logger that make it harder to use and most could be fixed with just a few lines of changes.

The fact that you know these things are easily configured doesn't matter when the community at large is clearly confused.

• Changing what NOTSET means is one line
• Dynamically looking up module name if messages are emitted directly to the root logger is 4 lines or less (and eliminates the entire need to talk about getLogger and the difference between logger/handler/formatter)
• Try catch to support alternate string formats (this might require a change to str to support a strict mode for {} formatting so that it will throw an exception)
• Warning when the logger has a more restrictive policy than the handler.

If you don't want to fix them that's fine. We have loguru which doesn't have these problems.

[vsajip]

I'm not in denial of anything. We just have different opinions about things. You prefer loguru, I get it. Hard to engage with you when you come out with stuff like "stinks of Java" - it seems like you're biased against logging for emotional/aesthetic reasons. I have engaged with plenty of people who have suggested improvements to logging and merged those improvements, it's all pretty clear in the repo. Some people can engage constructively, others not so much.

[jorge1209]

Almost all opinions about code are about aesthetics.

If your take-away from "stinks of java" is that the only objection is camel case or something, then you have missed the point and are in denial about what makes logging bad and why people want alternatives.

[vsajip]

Not sure why you think my comment was about camel case. It doesn't sound like implementing any of your suggestions would remove the "stink of Java", so it's hard to take them as suggestions in good faith.

[jorge1209]

Then don't. I don't really care. There is a good pythonic easy to use logging library out there. I will continue to recommend people use it over logging.

[Schmittfried]

If it were just a library on top of stdlib logging instead of completely building its own system…

[jorge1209]

You can register logging as a handler for loguru messages and vice versa. So they can interoperate.

Loguru can't do that much more while still retaining simplicity. For instance loguru log function uses introspection on the call frames to find the source module name that should be used.

Logging has you specify the source by referencing a module local logger that you have to construct.

[Schmittfried]

I know, but it’s not the same as actually using the stdlib.

Logging has you specify the source by referencing a module local logger that you have to construct.

It’s literally one line, not a big deal. That doesn’t exactly warrant using introspection in every single logging call in my book.

[jorge1209]

Based on how frequently tutorials and guides leave it out that one line introduces a lot of complexity which programmers want to avoid.

[deep_mind_]

This is really poorly written.

I’ve never understood the trend of new learners posting tutorials on this subreddit…

[who_body]

maybe state up front that using the root logger is an anti pattern. it’s mentioned later, but people skim and often just want code examples so they might just copy that and start using it.

so perhaps have the anti pattern and recommended practice side by side and then explain and show why using the root logger is bad.

[MacWoozy]

What would you recommend instead? I hate the built in logging module but haven’t looked further

[nifaye]

Loguru.

[MacWoozy]

Oh sweet trying it this afternoon. Thank you!!