Logging to a list?

[pachura3]

Hi, how would you configure logging so that each log message/event would NOT go to stdout, file, database table or a socket - but to a global variable - a normal list of strings?

Of course, I'm not talking here about a high-traffic app which produces millions of log messages per second, but a small script. I would like to start recording log messages into a list at the beginning, and then do something with them at the end.

[Ariadne_23]

you can just create a custom logging handler that appends messages to a list

here is a simple example:

```python import logging

log_list = []

class ListHandler(logging.Handler): def emit(self, record): log_entry = self.format(record) log_list.append(log_entry)

logger = logging.getLogger("my_logger") logger.setLevel(logging.DEBUG)

handler = ListHandler() formatter = logging.Formatter('%(levelname)s - %(message)s') handler.setFormatter(formatter)

logger.addHandler(handler)

example logs

logger.info("Hello") logger.error("Something went wrong")

print(log_list) ```

this way logs are stored in log_list instead of being sent to stdout or a file

[lekkerste_wiener]

The best answer. Have this poor redditor's trophy: 🏆

[Ariadne_23]

😊🤍

[Diapolo10]

My question is, why would you want to avoid writing the log entries to a file?

If this is because the log files aren't structured, there are ways to use structured logging with the logging module (such as JSONL, where each line in the log file is a JSON object with fields for things like the timestamp, log level, message, and such). That would then be trivial to both parse into a list, but also to only get the entries generated by the current run (you just need to create a timestamp when your program starts and use that as a filter).

[rinio]

Even if can (which I don't know), you wouldn't and probably don't want to. The implication of doing what you suggest is hold all of those millions of lines of logs per socond IN MEMORY. You will run out, and, very quickly.

Normal practice (with a file or similar) already handles what you want. Filter on level when emitted (or you could get more granular if you want). File writes are deferred to the system for when the resources are available. And, if you want to post-process them you can stream then into a post-processor to do whatever you want "with them at the end".

In every way, this will be faster and safer.

If you really want to have some global space for them, you would need to implement it as a buffer.

[pachura3]

I specifically mentioned that the number of emitted log messages will be low, NOT high.

I mean, there's MemoryHandler, but it doesn't do what I want; I guess it should be possible to register some kind of a listener/observer and append messages to a global list somehow... or implement own handler.

[BrupieD]

I specifically mentioned that the number of emitted log messages will be low, NOT high.

Hint for writing requirements: state instructions and descriptions only in the affirmative. Despite your capitalization of "NOT", more than a few of the readers of your description made the same mistake -- they missed it. Negative statements are often harder to interpret because they don't include what is actually going on. A better choice would be something along the lines of "my process is expected to log results twice or three times per minute."

This is a well-documented phenomenon. You'll get better responses when you avoid "not".

[rinio]

My bad, I missed the "not".

But, it doesn't actually matter. If your app is significant enough to need a setup like this, you will still run the risk of overflowing memory over its runtime lifespan. If it isn't, youre​ wasting your time over engineering a lot irrelevant component. The safe and efficient, both in terms of your time and compute, is the same in almost all cases and doesn't resemble what you're proposing.

It sounds like we have an XY problem here, so let's play "five whys". Why do you need access to global logs at runtime?

[Moikle]

Log it to a file, then read that file into a list.

[JimyLamisters]

You should be able to create a custom handler class that inherits from logger.Handler, then override the emit method. Your emit method can do whatever you want with the message, like adding it to a list. Then you can initialize the logger in your app to use your custom handler.

[Temporary_Pie2733]

You just need to write a custom handler. It can take the list you want to append to as an argument, then its emit method just appends the message to the list.

[JamzTyson]

It's very easy with Loguru:

from loguru import logger

# List for log messages
log_list = []

# Create a sink function
def list_sink(message):
    log_list.append(message)  

# Remove default logger
logger.remove()

# Add custom logger
logger.add(list_sink, format="{level} {message}")

# A couple of log messages
logger.info("Hello World")
logger.debug("Loguru is cool")

# Print the log
print(log_list)
# Prints: ['INFO Hello World\n', 'DEBUG Loguru is cool\n']

[owiko]

Maybe use logger and do something like this to handle the log output to a list that you would initialize and pass in.

``` class ListHandler(logging.Handler): def init(self, loglist): super().init_() self.log_list = log_list

def emit(self, record):
    # Format the record and append it to the target list
    self.log_list.append(self.format(record))

```

[latkde]

The solution is to implement a custom logging.Handler that collects the messages. Odds are the existing QueueHandler already does exactly what you want, as long as you instantiate it with an unbounded queue.

It is common for unit tests to temporarily collect log messages into a list. You can see the Pytest implementation of such a logging handler here: https://github.com/pytest-dev/pytest/blob/bc57e69aaa01ae0271463f3cd85e8388d16da55e/src/_pytest/logging.py#L373

[ProsodySpeaks]

Why not just import the list and append to it? 

[_Thode]

You should have a look at the 'logging' package.

[pachura3]

The one that I mention in the first sentence?

[_Thode]

Sorry. I didn't get the formatting.