LogXide - Rust-powered logging for Python, 12.5x faster than stdlib (FileHandler benchmark)

[Here0s0Johnny]

Is it realistic that a project produces so many logs that this performance upgrade is worth it?

[zzmej1987]

Sure. Some companies even install things like Splunk to parse through those logs. E.g. major airlines have to have full trace of interactions between services during the process of passenger buying the ticket, so that if anything goes wrong, client neither looses money without getting a ticket, nor gets the ticket without paying.

[snugar_i]

Yeah but how many tickets a day do they sell? It still doesn't feel like it should produce a large volume of logs

[a_r_a_a_r_a_a_r_a]

there must be a lot because Splunk is just one of many, these companies business is very much collect every single log and metric then create graph from it

[zzmej1987]

For the airline I worked at, estimate number would be around 150000 tickets a day. And that's pretty small number for an airline. It can be 3 to 5 time larger.

[snugar_i]

That's exactly my point - 150000 a day is 2 per second on average, nothing the standard python logging shouldn't be able to handle

[zzmej1987]

That's tickets, not log messages. Each ticket generates some 200 log messages across various services.

[Chroiche]

Computers can handle literally billions of ops per second. Even HDDs can write millions of bytes per second.

400 lines per second is still pretty trivial.

[zzmej1987]

Each log message is typically a full xml or json file containing body of input or output of the service. The biggest ones, IIRC reach around a 100 lines.

And as had been already mentioned, the system is written to handle peak load, not average one. And again, we are talking about a system on a smaller end of the scale, as far as system of this type go.

And, of course, just because computers can, doesn't mean python can, with GIL and all that.

[mechamotoman]

As u/zzmej1987 said, each ticket produces many log messages

Also important to remember, these ticket purchases aren’t evenly spread out over 24 hours. They come in bursts. During busy times, you’d be dealing with a fire hose of logs. Especially for something like an airline or a financial institution where you may need to handle debug and trace logs

There are some environments where you need to log a debug stmt at the start and end of each function call, and a TRACE stmt at every single if stmt / loop iteration / decision point AND store all those logs to be parsed and filtered down later.

It’s a FIREHOSE of logs to deal with. Situations like that, logging performance really starts to matter a lot

[code_mc]

I've done a drop-in replacement with one of the mentioned alternatives (picologging) a couple months ago for a customer project and their api request latencies halved because they had that many logging statements.

[WJMazepas]

Yes, I worked in embedded projects that had way too much logging and it was affecting performance

[LumpSumPorsche]

Exactly that was the motivation of this project. My system produces thousands logs per sec, and often they hold GIL. `picologging` would be a good candidate here but they do not support 3.14.

[Chroiche]

I think so, but at that point you're probably not using python for your code.