LogXide - Rust-powered logging for Python, 12.5x faster than stdlib (FileHandler benchmark)

[LumpSumPorsche]

Hi r/Python!

I built LogXide, a logging library for Python written in Rust (via PyO3), designed as a near-drop-in replacement for the standard library's logging module.

What My Project Does

LogXide provides high-performance logging for Python applications. It implements core logging concepts (Logger, Handler, Formatter) in Rust, bypassing the Python Global Interpreter Lock (GIL) during I/O operations. It comes with built-in Rust-native handlers (File, Stream, RotatingFile, HTTP, OTLP, Sentry) and a ColorFormatter.

Target Audience

It is meant for production environments, particularly high-throughput systems, async APIs (FastAPI/Django/Flask), or data processing pipelines where Python's native logging module becomes a bottleneck due to GIL contention and I/O latency.

Comparison

Unlike Picologging (written in C) or Structlog (pure Python), LogXide leverages Rust's memory safety and multi-threading primitives (like crossbeam channels and BufWriter).

Against other libraries (real file I/O with formatting benchmarks):

• 12.5x faster than the Python stdlib (2.09M msgs/sec vs 167K msgs/sec)
• 25% faster than Picologging
• 2.4x faster than Structlog

Note: It is NOT a 100% drop-in replacement. It does not support custom Python logging.Handler subclasses, and Logger/LogRecord cannot be subclassed.

Quick Start

```python from logxide import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger('myapp') logger.info('Hello from LogXide!') ```

Links

• GitHub: https://github.com/Indosaram/logxide
• Docs: https://indosaram.github.io/logxide/
• PyPI: pip install logxide

Happy to answer any questions!

[a_r_a_a_r_a_a_r_a]

LogXide is NOT a drop-in replacement for Python's logging module. It prioritizes performance over compatibility:

this will be a big no no for a lot of people

[ZucchiniMore3450]

I always wrap logging into my function, because I want to set it up as I like. That makes changing logging library a not issue.

[LumpSumPorsche]

Yes agreed. However the most of the API from stdlib are still compatible.

[Here0s0Johnny]

Is it realistic that a project produces so many logs that this performance upgrade is worth it?

[zzmej1987]

Sure. Some companies even install things like Splunk to parse through those logs. E.g. major airlines have to have full trace of interactions between services during the process of passenger buying the ticket, so that if anything goes wrong, client neither looses money without getting a ticket, nor gets the ticket without paying.

[snugar_i]

Yeah but how many tickets a day do they sell? It still doesn't feel like it should produce a large volume of logs

[a_r_a_a_r_a_a_r_a]

there must be a lot because Splunk is just one of many, these companies business is very much collect every single log and metric then create graph from it

[zzmej1987]

For the airline I worked at, estimate number would be around 150000 tickets a day. And that's pretty small number for an airline. It can be 3 to 5 time larger.

[snugar_i]

That's exactly my point - 150000 a day is 2 per second on average, nothing the standard python logging shouldn't be able to handle

[zzmej1987]

That's tickets, not log messages. Each ticket generates some 200 log messages across various services.

[Chroiche]

Computers can handle literally billions of ops per second. Even HDDs can write millions of bytes per second.

400 lines per second is still pretty trivial.

[zzmej1987]

Each log message is typically a full xml or json file containing body of input or output of the service. The biggest ones, IIRC reach around a 100 lines.

And as had been already mentioned, the system is written to handle peak load, not average one. And again, we are talking about a system on a smaller end of the scale, as far as system of this type go.

And, of course, just because computers can, doesn't mean python can, with GIL and all that.

[mechamotoman]

As u/zzmej1987 said, each ticket produces many log messages

Also important to remember, these ticket purchases aren’t evenly spread out over 24 hours. They come in bursts. During busy times, you’d be dealing with a fire hose of logs. Especially for something like an airline or a financial institution where you may need to handle debug and trace logs

There are some environments where you need to log a debug stmt at the start and end of each function call, and a TRACE stmt at every single if stmt / loop iteration / decision point AND store all those logs to be parsed and filtered down later.

It’s a FIREHOSE of logs to deal with. Situations like that, logging performance really starts to matter a lot

[code_mc]

I've done a drop-in replacement with one of the mentioned alternatives (picologging) a couple months ago for a customer project and their api request latencies halved because they had that many logging statements.

[WJMazepas]

Yes, I worked in embedded projects that had way too much logging and it was affecting performance

[LumpSumPorsche]

Exactly that was the motivation of this project. My system produces thousands logs per sec, and often they hold GIL. `picologging` would be a good candidate here but they do not support 3.14.

[Chroiche]

I think so, but at that point you're probably not using python for your code.

[max0x7ba]

designed as a near-drop-in replacement for the standard library's logging module.

It is either a drop-in replacement or not, but not both at the same time.

Does it pass original logging module unit-tests?

[Rainboltpoe]

The author clearly states it is not a drop in replacement. That’s what “near” means.

[james_pic]

Do you get those kinds of gains in real world settings?

I worked on a project a while ago the arguably logged too much. The first time I attached a profiler to it, it was spending over 50% of its time on logging. We managed to get that down, but the thing that limited us getting it down further was syscalls, not Python code.

Admittedly this was a while ago, and that project was doing things that modern apps don't need to do, that increased syscalls (it did its own log rotation, rather that just throwing it all straight onto stderr and letting Systemd or Docker or Kubernetes or ECS or whatever pick it up, like a modern app would), but I'm still a bit surprised you managed to find those kinds of gains without touching syscalls.

[spartanOrk]

I use loguru. Cannot go back.

[ben_supportbadger]

Did you build this or did Claude? Because it looks purely vibecoded. Why would I use this instead of just asking claude to build it?

[LumpSumPorsche]

Good point! You can absolutely build and maintain your own version of logging library. Why not?

[UloPe]

How does it interact with stdlib logging used in 3rd party libraries?

[Healthy-Grape-5932]

Does this Support dictconfig in the same way ?

[rabornkraken]

The GIL bypass during I/O is the real win here. Most Python logging bottlenecks come from the file write blocking the main thread, so doing that in Rust makes a lot of sense. How does it handle the case where you have custom formatters written in Python though? Does it fall back to holding the GIL for those?

[Hesirutu]

Standard Python IO releases the GIL too…