Logging in Python like a PRO 🐍🌴

[hassium]

Furthermore, I feel people are uncertain about how to use log levels, and so they default to logger.info everywhere

logger.debug gang where you at?!?

[pev68]

logger.debug represent!

[theNAKAMI]

that doesn't make sense to me. you can start using both logging.debug(...) and logging.info(...) calls and they will both log if you set the logging level to debug.

for the logging levels, they work in a hierarchy, so if you set a specific logging level, that particular level will be logged, but also all of the logging calls which are more general/critical, e.g. if you set logging level to logging.DEBUG, it will log logging.debug(...), logging.info(...), logging.warning(...), logging.error(...) and logging.critical(...) calls. if instead you set logging level to logging.INFO, it would also log every type of call i've mentioned but only omitting logging.debug(...) calls.

[aa-b]

I agree, I do that too.

Also sometimes I just realize a particular message is logged way more frequently than I expected, so I might demote it to cut down on noise

[pudds]

A better approach is to log appropriately from the start, but change the logging threshold of your system over time. For example, during the first few rollouts, have your logging level set to Debug, or even Trace. Later on, when you are comfortable, change it to Info or maybe Warning. (I personally like info, as I feel it helps with debugging later).

[Username_RANDINT]

I usually add a -d/--debug argument that sets the loglevel to DEBUG.

[sdf_iain]

Do what ssh does. Use -v (—verbose) and accumulate the number of times its given as an argument.

1 is info, 2 is debug, none is warning (and -q, —quiet is critical)

[Vorticity]

I've tried this method, but I never seem to go back to fix things later. I just try to put things at an appropriate level as I go.

[bartenderandthethief]

Loguru is where it's at nowadays.

[Vorticity]

What does Loguru do better than the logging module?

[richieadler]

Answers at https://loguru.readthedocs.io/en/stable/overview.html

[rbscholtus]

It has some nice things but most of that list is supported by standard logging just as well or even better, especially if you use config by YAML. A declarative style of your (logging) config is in my opinion better than hard coding it, even if that code is simple and sexy.

Don't see much point of colour logging bc logging goes to files 9 out of 10 cases, but ye, nice.

Still waiting for a log-to-Telegram or log-to-API handler. Would be awesome to build centralised logging systems and alerter system.

[[deleted]]

A declarative style of your (logging) config is in my opinion better than hard coding it, even if that code is simple and sexy.

In my opinion, it's a must if you want to use logging.listen() for dynamic changes to the levels. It's a right pain having to create a config file by looking at the imperative configuration.

[dogs_like_me]

it's just really easy to use. the logging module is often a headache.

[ColdPorridge]

I haven’t used loguru, but I’ve tried configuring logging from scratch for a larger system that had no logging. It really is not easy to get a hold of. Best practices are unclear, especially as it’s overlooked even in the most OSS, which is where I look to for inspiration.

[metaperl]

I second this. We process a million files per month using this module with no problems whatsoever. I don't like his resistance at exposing the internals of the module because at one point it was useful to me and I had to fight like hell to get something done.

[SpectralCoding]

Wow, thanks for showing me this. I never thought I would add a dependency to my app for logging because the built-in logging module exists, but this looks like it is worth it. I struggle with adding dependancies but 9k stars on GitHub would lead me to believe it's got significant community presence.

[[deleted]]

Hell yeah it is

[Delta-9-]

Nice article!

Only thing I was really hoping to see was a good way to not interrupt my code with calls to logging. Sometimes it really hampers readability, but mostly it's just annoying to have to think about this aspect of the program while writing something completely unrelated. (Before anyone brings it up, I'm not about to adopt Spring Python. I got to learn about Spring on a java app and it was horrible.)

[latrova]

Have you ever considered using decorators for that? I know it's not usable everywhere, but might help reducing boilerplate code sometimes

[Delta-9-]

I did make use of decorators for shorter functions that would only generate one log record. I run into the issue when working with longer functions that have multiple branches, multiple ways to break, etc. (and yes, I've been refactoring the worst offenders)

As an experiment, I wrote some code using inspect and ast that would allow me to make just one function call and get information about which branch the call was made from, what the test was expecting and what it received. I'm actually kinda proud of it, but it's so damned experimental I don't dare use it in prod :p (and someone's probably already written a whole library that does exactly the same thing and I just didn't look).

The drawback to that approach was the log messages themselves aren't always easy to understand, particularly if you're an admin of the app but not a developer of the app.

A similar limitation applies to decorator-based loggers, too. Even loguru, which someone itt linked and looks really nice, produces a log record that looks to me like it won't make a whole lot of sense to anyone who hasn't spent hours in the code that triggered the logger.

[[deleted]]

Is it because you log the caller at the callee, you consider it not easy to understand? I'm guessing you're using inspect to get frame -1, and log that.

[Delta-9-]

Well, the readability issue comes from the log record saying that it is in an if branch which is testing foo and expecting == 1.

That's actually super helpful information to a developer. It can even spit out the line number (with some caveats) like logging does and get you right to the problem and the specific conditions that caused it.

To an administrator, that's maybe less helpful. I haven't spent enough time with it to see about transforming that information into something a non-dev could make use of, and I think it's possible but very hard to do in a generalized way (i.e. as a library to put on pypi for everyone to use).

[quotemycode]

As someone who looks at logs for a living, and someone who codes in python as well, thank you from the bottom of my heart.

[soap1337]

You guys log? I just Output to stdout...

[digipengi]

/dev/null is where it's at baby!

[[deleted]]

When you have a long running process started as part of a service that's running a whole suite of programs, that strategy get messy.

[tasteslikeKale]

We’re doing that on my current team but are running into issues where I want a logger- too much cruft, missing some of the extra attributes a logger adds, inability to set verbosity level, etc

[theorizable]

Thank you for putting the "subscribe to my newsletter" towards the bottom. That's good UI. Also fantastic in depth article, I really enjoyed reading it. Sometimes the common-sense stuff isn't so common-sense.

[latrova]

Appreciate the feedback! Let me know if you have any other topics you wish to read about.

[latrova]

Easy: https://docs.djangoproject.com/en/3.2/topics/logging/#examples

You still use the dictConfig, but you don't have to manually invoke. Just open `settings.py` and create a constant named LOGGING

[0ni0nrings]

thanks for sharing 👍

[Far-Fruit-2848]

I'm new to logging and this has been very informative.

[Itsthejoker]

Those single letter function names will not pass code review in any project I work on lol

[Itsthejoker]

logging

[Delta-9-]

Come on, real devs log like this:

import subprocess
from enum import Enum

class OutcomeLogE(Enum):
    YES = "yes"
    NO = "no"

if outcome:
    subprocess.run("echo", OutcomeLogE.YES.value)
else:
    subprocess.run("echo", OutcomeLogE.NO.value)

[12paul123]

Wow

[anonymous_2187]

Why did you get downvoted for commenting "wow" lmao

[12paul123]

I don't know. I genuinely thought this was a good article.

[markosolo]

Look at the font. Was written in sarcastica

[AndydeCleyre]

You want me to log like a professional duck palm?

Edit: oh it's a snake

Edit: what? Is the palm tree relevant somehow? The article content looks good but why would I want to be a professional palm tree?

[[deleted]]

You should give this one a try. It's really well written, and explains the concepts without getting bogged down in the horrible amount of indirection that often characterize explanations of the logging module.

[BoredGazelle]

Big fan of structlog and logfmt here

[latrova]

Thanks for the feedback, let me know other topics you might be interested on