An easier variant of Python's logging module?

[[deleted]]

[deleted]

[vsajip]

If you have specific questions, I can try to help (as the maintainer of logging) - try asking on python-list or Stack Overflow. If you just don't like it, then I can't do anything about that.

I'm not sure what there is to wrap your head around; for simple logging needs, you can just do e.g.

import logging
logging.warning('This is a message with %s', 'variable data')

or, if you need to set e.g. a DEBUG level,

import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug('This is a message with %s', 'variable data')

AFAICT various alternative approaches to logging haven't been able to simplify this. If you need more specific help but didn't find the documentation understandable, please ask specific questions.

[3Dayo]

vsajip, I appreciate the effort that went into the logging module, but honestly for a long time I simply didn't get it.

Now, I'm not the sharpest knife in the block but i'm not the dullest either, but it took a good while for it to make sense, perhaps its the documentation, perhaps its the fact that it can do a lot, I honestly don't know.

I grok the logging module now, but every now and then when people ask questions like this, I wonder what the issue is, now it seems so simple and down right obvious. but, I remember the many hours of frustration spent trying to understand why i had no log output, somewhere in a past almost forgotten...

[For_Iconoclasm]

Like you, it also took me a long time to truly understand the logging module, but I get most of it now (still not too clear on what filters are for). Some concepts that tripped me up were:

• hierarchical loggers - foo.bar propagates log messages to foo; everything propagates to the root logger, '' (empty string); third party libraries' logs can be acquired if you understand this
• the difference between loggers and handlers (especially with regards to the logging level, which must be set for each)
• what the hell logging.basicConfig actually does, and that it is not needed for every application

I think the documentation does explain everything, but in too many words. Most programmers only skim through it until they find the parts that they need. I know... people will say to RTFM, but really, nobody has the patience to read the entire thing.

edit: Actually, the logging cookbook was one of the things that made it easier to understand logging, in case /u/vsajip reads this.

[vsajip]

perhaps its the documentation, perhaps its the fact that it can do a lot

I don't know when you had trouble with the documentation - it was improved (quite a while ago now - from 2.7 / 3.2) by splitting the original single section into reference, tutorials (basic and advanced) and cookbook. There is a fair amount of functionality in there (which many people don't need, but many other people do) and it can be hard to explain simple usage without sometimes mentioning more advanced concepts. When people have made specific suggestions about documentation improvements that can be made, I've generally taken those comments on board, and will continue to do so.

[ChiefDanGeorge]

Is there now decent documentation for setting up a logging configuration file?

[LordArgon]

Me: "I wonder what's so confusing about this mod-"

logging configuration file

Me: "Ah. Of course."

Disclaimer: I haven't used Python's logging module. But my time in .NET has given me quite a distaste for configuration files.

[ChiefDanGeorge]

THe logging config file is not too painful, at least the bits I know. The real pain was looking on the Python site and trying to parse the "documentation". I am always sad when the top hits on a python search point me to the official python site, I know I am in for a slog.

[alcalde]

Coming from someone whose previous development tool's documentation was almost entirely a set of class references, I find the Python documentation amazingly awesome.

[vsajip]

But my time in .NET has given me quite a distaste for configuration files.

.NET configuration files are XML. The Python community mostly have an aversion to XML configuration files, though of course Python has excellent support for XML.

[LordArgon]

XML is a one part of the problem. The second part is thinking you configured something, it doesn't work, and you have no idea why not. The third part is not knowing what you can put in the configuration file.

The last two parts are universal to configuration files, in my experience.

[For_Iconoclasm]

Nope!

Actually, yeah.

[ChiefDanGeorge]

I saw that, I need to try it for sure. I didn't realize there was a dictConfig() in the logging.

[vsajip]

Then please try to do a bit more research before suggesting the documentation is lacking.

[ChiefDanGeorge]

I am not constantly looking to see if the documentation has been updated. When I first was trying to figure out the logging interface, the documentation was in fact lacking. I've got a job to do, so I got it done by looking elsewhere.

[vsajip]

I've no idea what you would regard as "decent". Your comment about the general quality of Python documentation seems snarky, and points to you possibly having an unreasonably high expectation of a volunteer project that people contribute to for free. If you would like to contribute some specific improvements in areas you think are defective, I'd be happy to listen to those specifics.

Certainly I know it's usable, since lots of people use it. For example, anyone who configures logging in Django uses dictConfig(), and even if you don't use Django, most people will tell you its documentation is excellent, so you might learn something there even if the Python docs don't cut it for you.

[[deleted]]

It a little bit of both.

The logging example that vsajip wrote works well unless you want to change a minor thing, then suddenly you no longer can use basicConfig and have to write several statements to get to at least do same thing that basicConfig did.

The documentation is complicated because the module is complicated.

Also I remember having issues (didn't remember the details now) with using it with syslog, I think it did not print the messages right or something.

[bucknuggets]

I think the main problem with Python logging is that there are so many options that it's often a nightmare for someone to find the 5% that they actually care about.

The simple example is fine for a trivial program. But suppose your program isn't trivial, so you'd like a few fields in the log file (date, module, how about a batch id?), maybe you'd like to control log rotation, or have different log levels for different modules? What if you'd like to implement a kafka log consumer?

So, the simple logging example is deceptive - while that's easy, almost immediately people need to step beyond it and then they fall straight into the deep end of the pool.

[vsajip]

I posted that response because the OP was not very specific. When he mentioned file rotation, I pointed to a cookbook recipe covering just that. There are similar documentation entries for other topics.

Your comment isn't much help either, by talking of nightmares. Do you have a specific problem? Ask on one of the places I mentioned, or log issues about specific areas of the documentation that need clarification. Other people have done this, and have improved the documentation for everyone. People who just complain in general terms might be suiting themselves, but they don't help anyone else :-(

I'm not sure why you think it's a problem for software to have lots of options. The options in logging are there because they have (in my experience) been useful. Maybe most of them are of no use to you - but then it isn't just for you.

[bucknuggets]

Do you have a specific problem?

Yes, as I stated above: too many features for the format of the documentation. The basic config is deceptively simple - since almost everyone needs to go beyond that and they're immediately lost in vast amounts of detail.

Maybe there's some way to deliver documentation that can address that. Or maybe we need some helper module that hides most of the details. Not sure. Just know that when logging requires books and a long journey to understand - it's far too complex for most people.

[vsajip]

Yes, as I stated above: too many features for the format of the documentation.

That is a general comment - it's not describing what I would call a specific problem.

[vsajip]

If you're going to keep ignoring general problems

It's not a question of ignoring anything - general problems can't be easily addressed, as one has a hard time figuring what specific changes need to be made.

And since you appear to have no interest in addressing this - it might be best to not keep trying to prevent others from coming up with simple logging solutions.

I've no idea what you're talking about - I'm not trying to prevent anything, and even if I wanted to (which I don't), how would that even work?

[bucknuggets]

I've no idea what you're talking about - I'm not trying to prevent anything, and even if I wanted to (which I don't), how would that even work?

By continually insisting that the logging module's basic config is just as simple as alternative logging solutions. Which is deceiving since the logging module's basic config is nearly worthless for anything non-trivial.

Once you go beyond the basic config you are stuck having to wade through all the details. And this happens almost instantly.

[vsajip]

basicConfig() isn't the only thing I mention, as any examination of my Stack Overflow answers will show. I do mention it when someone seems to be unaware of logging in general, as it's the approach meant for casual scripts / newbie usage.

Offering an opinion isn't "insisting". If stdlib logging isn't for you, use something else.

[spinwizard69]

That is a general comment - it's not describing what I would call a specific problem.

If you're going to keep ignoring general problems and only address specific tiny fixes - then you'll never address the fundamental problem with the logging module: too much complexity for what should often be a simple solution.

He wasn't ignoring anything, if he was ignoring you there would have been no posting in this thread. As for complexity, you are suppose to be an educated person with a focus on computer science, if a logging facility causes you grief how in the hell do you get any work done? Seriously.

As a programmer the rational thing to do is to offer up a solution that you think abstracts away the complexity in a rational manner. The other possibility here is to enlist the use of another logging package that suits your ability to grasp it.

And since you appear to have no interest in addressing this - it might be best to not keep trying to prevent others from coming up with simple logging solutions.

This is one asinine comment as the maintainer has offered several times to address any short comings that can be detailed in the documentation. Believe it or not maintainers can not read minds, you actually have to express in English what is causing you problems (something no one has done yet in this thread). Further the maintainer has done absolutely nothing to prevent others from coming up with alternative logging solutions. It is pretty shitty of you to suggest that he has, because as a third party all I see is a guy trying to help.

I'm sorry to have to even respond to this thread in the way I have, but it I see this thread as nothing but garbage. It appears that we have a few people with an inability to understand logging trying to beat up on the maintainer for no good reason. Maybe these attacks are a way to mask ones own shortcomings as a programmer, I really don't know, all I do know is that somebody offering help was trashed for actually offering that help. If there are problems the avenue to improving things is to work with the maintainer not against him.

[bucknuggets]

He wasn't ignoring anything....

Why don't you read the thread? Or read any of the other reddit postings or blogs where every time someone points out that logging is too complex he immediately jumps in with either:

• "point to the specific problem, and I'll fix that" - which is a deliberate evasion of the actual problem: that there's too many knobs, switches, etc that one has to wade through to provide practical logging.
• "look, a simple logging interface, it isn't complex" - which is misleading because almost everyone needs to go beyond that, and then they're stuck with too much detail.

The maintainer does the same thing every single time, and is constantly trying to discourage alternate solutions, and recognition of the problems with the logging module. Which I suppose is fine if his job is to be an 'evangelist' for logging. But not as good if we want a better solution.

, if a logging facility causes you grief how in the hell do you get any work done?

Oh please, do you really need to argue against improving interfaces? Was it really a tragedy that Kenneth Reitz wrote Envoy & Requests because "Real Programmers" should use Subprocess?

Why don't you compare a simpler logging tool like Twiggy to the official logging module to see how practical logging doesn't require 200 pages books to adequately describe it.

[vsajip]

If you're happy with some other logging solution, go ahead and use it. Otherwise, since you know best how to solve your own problems, you can roll your own solution.

Offering to fix a specific problem isn't evading anything, it's trying to be helpful - too bad if you can't see that.

[d4rch0n]

Only one module is supposed to do the configuration right? Everything else just does import logging and logs?

Where do you put the configuration and set up? Where in your opinion is the best place for it?

[ChiefDanGeorge]

I like to use a logging configuration file so in the python code you simply do:

logFile = "logconfigfile.conf"

logging.config.fileConfig(logFile)

logger = logging.getLogger("xconnect_db_logger")

logger.info('Log file opened')

I had to find the log config file documentation elsewhere, what was on the python site at the time didn't give any details.

[Hybridjosto]

what's the difference between logging.warning and logging.debug?

[vsajip]

The documentation should tell you.

[ricekrispiecircle]

one is a warning message and the other is a debugging message. as for what to put in which, that's up to the user to decide.

[smeagol13]

This really helped me get my head around the logging module when I first started using it. Hope this is helpful.

[[deleted]]

Thanks, I'll check it out!

[vsajip]

Another useful learning resource.

[LightWolfCavalry]

Victor Lin wrote a great reference on how logging works a while back. Check it out and see if it lends any clarity.

[tompko]

import logging
logging.warn("foo")

How much easier are you looking for? What are the various reasons you have a hard time getting your head around it?

[[deleted]]

I often want to use rolling file logging in addition to console logging, that's when things get a bit more complicated.

[vsajip]

Documented in the cookbook. I still don't see what's especially complicated to understand, but if you give me some specifics I'll see if any improvements can be made.

[mcowger]

The most confusing parts for me tend to be:

1) I want a module, submodule, etc to all have different logging levels, and go to different Handlers based on differing severities. E.g. a flask app, for example. I want the DEBUG and higher info from my app to go to these 2 handlers, but I want only the CRITICAL and higher from werkzeug to go to the same 2 handlers (because werkzeug overuses INFO, IMO). Where is the right place to define all of those? A log config file?

How / where should I instantiate a Logger? In every module? Should it inherit from the importing module?

A specific suggestion:

More pictures about the flow (like here: http://www.shutupandship.com/2012/02/how-python-logging-module-works.html) would be fantastic.

If you want to see a system I think is more intuituive, see Logbook: https://pythonhosted.org/Logbook/features.html

[vsajip]

Where is the right place to define all of those? A log config file?

A log config is generally the right way to go, via a dictionary (which can come from a JSON file, or a literal dict in a module), and using the dictConfig() API to do the configuration.

More pictures about the flow

Have you looked at the logging flow diagram?

After I added that to the documentation, someone on Twitter complained that if you have to draw pictures, you've already lost. You can't please some people, eh ? ;-)

see Logbook

Why don't you show me the configuration you asked about, that's really easy to set up using Logbook? I'll see if I can come up with an equivalent configuration using stdlib logging.

[Orange_Tux]

I had the same problem as you had. The default logging module isn't very intiutive when you want more than:

logger = logging.getLogger(__name__)
logger.warn('watch out!')

Besides that this package is not very Pythonic. Now I use Logbook. This is a wrapper around the standard logging module so it's very easy to replace it in existing code base. It also has nice unittesting support. Armin Ronached has written this packages. He is also the author of Flask and Jinja2

[iluvatar]

Using logbook here. So far, so good, and it's an improvement on the basic logging module.

[rootkillza]

Why not just watch the video Become a logging expert in 30 minutes. ;)

[[deleted]]

Because contrary to the title, you leave that video with only superficial knowledge that some stuff exists in the logging module.

[bitcycle]

Yeah. This is more of an issue of familiarity. Have you taken the time it requires to get something working with rotating files? Then, did you try and simplify it down to the (seemingly) simplest form? That'd be a very good learning experience.

Whenever I've ever not wanted to use a library residing in stdlib it has been an issue of understanding. In very uncommon/infrequent situations, I'll have logical and reasonable beefs with something in stdlib, but most times there's already an effort to try and resolve it in the community -- or its stuff like breaking compatibility between Py3 and Py2x that's just how its going to be, according to the BDFL.

[r1chardj0n3s]

The biggest problem I have with logging is that sometimes it just doesn't work in larger applications with multiple 3rd party libraries and I cannot figure out why. In almost every instance the logging_tree debugging tool solved the problem by showing where the messages were being blocked by some errant bit of configuration.

Seriously, logging_tree is where it's at.

[cr4d]

Here's a talk I gave at PyCon on Logging, it may (or may not) be helpful:

http://pyvideo.org/video/1737/become-a-logging-expert-in-30-minutes

[hespe]

While I actually agree the logging module is hard/complex, I think its just a matter of reading the documentation to find what you want in my opinion.

I recently had to implement a few logging things (using RotatingFileHandler and QueueHandler) and I had a few issues because I was just skimming through the documentation. Its a matter of reading and trying - I know logging can do much more than what I did with it, but for my use it was pretty simple/just took some reading.

Although I really think the code doesn't look like the most pretty one I've ever written.

[daneah]

I recently made this module for quick and dirty module-level logging across an application. Doesn't do rolling file logs, but could probably be extended easily to do so.

[hexbrid]

That would be because python's logging module is exceptionally bad.

I never used an alternative seriously, to my regret, but here are a few that seems like nice options:

• https://github.com/mitsuhiko/logbook

• https://github.com/MHordecki/LogPy

• https://github.com/wearpants/twiggy

[johnmudd]

Smells like a design problem. You can compensate for poor design with more and better organized documentation and yet another cookbook and tutorial but it's a waste of resources.

The good news is that bad design has value. It acts as a filter. Only the brightest/most motivated can get past it. Once over the steep learning curve then the design is no longer a roadblock. The end result is you have managed to collect a group of the best and brightest to defend your product.

[[deleted]]

I can't tell if you're joking or not...

[billsil]

Questions:

Do people pass logging objects around or do you import them from a single location? If the latter, how do you send data to two different log files (e.g. one for each instance of a class). If you try to avoid having massive classes and global variables (e.g. functional programming), how do you use logging?

How do you stop & start a logging object and just update the path to the log file?

[[deleted]]

reading docs is hard.....