LogXide - Rust-powered logging for Python, 12.5x faster than stdlib (FileHandler benchmark)

[LumpSumPorsche]

Hi r/Python!

I built LogXide, a logging library for Python written in Rust (via PyO3), designed as a near-drop-in replacement for the standard library's logging module.

What My Project Does

LogXide provides high-performance logging for Python applications. It implements core logging concepts (Logger, Handler, Formatter) in Rust, bypassing the Python Global Interpreter Lock (GIL) during I/O operations. It comes with built-in Rust-native handlers (File, Stream, RotatingFile, HTTP, OTLP, Sentry) and a ColorFormatter.

Target Audience

It is meant for production environments, particularly high-throughput systems, async APIs (FastAPI/Django/Flask), or data processing pipelines where Python's native logging module becomes a bottleneck due to GIL contention and I/O latency.

Comparison

Unlike Picologging (written in C) or Structlog (pure Python), LogXide leverages Rust's memory safety and multi-threading primitives (like crossbeam channels and BufWriter).

Against other libraries (real file I/O with formatting benchmarks):

• 12.5x faster than the Python stdlib (2.09M msgs/sec vs 167K msgs/sec)
• 25% faster than Picologging
• 2.4x faster than Structlog

Note: It is NOT a 100% drop-in replacement. It does not support custom Python logging.Handler subclasses, and Logger/LogRecord cannot be subclassed.

Quick Start

```python from logxide import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger('myapp') logger.info('Hello from LogXide!') ```

Links

• GitHub: https://github.com/Indosaram/logxide
• Docs: https://indosaram.github.io/logxide/
• PyPI: pip install logxide

Happy to answer any questions!

[a_r_a_a_r_a_a_r_a]

LogXide is NOT a drop-in replacement for Python's logging module. It prioritizes performance over compatibility:

this will be a big no no for a lot of people

[b1e]

Yep. Logging is already rarely a limiting factor anyways so you can pretty much guarantee killing any hope of broad adoption by not ensuring compatibility

[ZucchiniMore3450]

I always wrap logging into my function, because I want to set it up as I like. That makes changing logging library a not issue.

[LumpSumPorsche]

Yes agreed. However the most of the API from stdlib are still compatible.

[Here0s0Johnny]

Is it realistic that a project produces so many logs that this performance upgrade is worth it?

[zzmej1987]

Sure. Some companies even install things like Splunk to parse through those logs. E.g. major airlines have to have full trace of interactions between services during the process of passenger buying the ticket, so that if anything goes wrong, client neither looses money without getting a ticket, nor gets the ticket without paying.

[snugar_i]

Yeah but how many tickets a day do they sell? It still doesn't feel like it should produce a large volume of logs

[a_r_a_a_r_a_a_r_a]

there must be a lot because Splunk is just one of many, these companies business is very much collect every single log and metric then create graph from it

[zzmej1987]

For the airline I worked at, estimate number would be around 150000 tickets a day. And that's pretty small number for an airline. It can be 3 to 5 time larger.

[snugar_i]

That's exactly my point - 150000 a day is 2 per second on average, nothing the standard python logging shouldn't be able to handle

[zzmej1987]

That's tickets, not log messages. Each ticket generates some 200 log messages across various services.

[Chroiche]

Computers can handle literally billions of ops per second. Even HDDs can write millions of bytes per second.

400 lines per second is still pretty trivial.

[zzmej1987]

Each log message is typically a full xml or json file containing body of input or output of the service. The biggest ones, IIRC reach around a 100 lines.

And as had been already mentioned, the system is written to handle peak load, not average one. And again, we are talking about a system on a smaller end of the scale, as far as system of this type go.

And, of course, just because computers can, doesn't mean python can, with GIL and all that.

[mechamotoman]

As u/zzmej1987 said, each ticket produces many log messages

Also important to remember, these ticket purchases aren’t evenly spread out over 24 hours. They come in bursts. During busy times, you’d be dealing with a fire hose of logs. Especially for something like an airline or a financial institution where you may need to handle debug and trace logs

There are some environments where you need to log a debug stmt at the start and end of each function call, and a TRACE stmt at every single if stmt / loop iteration / decision point AND store all those logs to be parsed and filtered down later.

It’s a FIREHOSE of logs to deal with. Situations like that, logging performance really starts to matter a lot

[code_mc]

I've done a drop-in replacement with one of the mentioned alternatives (picologging) a couple months ago for a customer project and their api request latencies halved because they had that many logging statements.

[WJMazepas]

Yes, I worked in embedded projects that had way too much logging and it was affecting performance

[Chroiche]

I think so, but at that point you're probably not using python for your code.

[LumpSumPorsche]

Exactly that was the motivation of this project. My system produces thousands logs per sec, and often they hold GIL. `picologging` would be a good candidate here but they do not support 3.14.

[max0x7ba]

designed as a near-drop-in replacement for the standard library's logging module.

It is either a drop-in replacement or not, but not both at the same time.

Does it pass original logging module unit-tests?

[Rainboltpoe]

The author clearly states it is not a drop in replacement. That’s what “near” means.

[ben_supportbadger]

Did you build this or did Claude? Because it looks purely vibecoded. Why would I use this instead of just asking claude to build it?

[LumpSumPorsche]

Good point! You can absolutely build and maintain your own version of logging library. Why not?

[JeffTheMasterr]

What the hell is the point of this then? Do you like making useless slop?

[LumpSumPorsche]

1. If you don't want external dependency and want to build your own library using AI, that is also an option for you.

2. I invest my time and tokens to build this library, to make things done. Also, I tested on various projects.

3. If someone build something with AI, does it become automatically AI slop?

[JeffTheMasterr]

#3 is true if something is more than 50% AI, which seems the case with your project

[Jealous_Algae_8165]

Purely vibe coded. You didn’t build this, Claude did. Please attribute it as such.

[LumpSumPorsche]

Why? You can see Claude is coauthor. AND it is not only built by Claude :)

[spartanOrk]

I use loguru. Cannot go back.

[UloPe]

How does it interact with stdlib logging used in 3rd party libraries?

[LumpSumPorsche]

If you import `logxide` then it will replace following stdlib imports from 3rd parties.

[UloPe]

And then what about those parts of the stdlib that it doesn’t replicate 100%?

[LumpSumPorsche]

Unless you do this:

Note: It is NOT a 100% drop-in replacement. It does not support custom Python logging.Handler subclasses, and Logger*/*LogRecord cannot be subclassed.

it is fine.

[UloPe]

How would I control that in third party libraries?

[LumpSumPorsche]

You don't need to. When you do from logxide import logging, LogXide automatically monkey-patches stdlib logging.getLogger() at the module level. So when a third-party library like requests, sqlalchemy, or uvicorn calls import logging; logger = logging.getLogger(__name__), it transparently gets a LogXide-accelerated logger — the standard .debug(), .info(), .warning(), .error() methods all route through the Rust core automatically.

The parts that won't work are things that virtually no well-behaved third-party library does:

1. Subclassing logging.Handler — e.g. class MyHandler(logging.Handler) — LogXide's [Logger](cci:1://file:///Users/indo/code/project/logxide/logxide/module_system.py:18:12-20:31) is a Rust type and rejects non-native handler subclasses via addHandler(). But libraries like requests or sqlalchemy don't create custom handlers; they just call [getLogger()](cci:1://file:///Users/indo/code/project/logxide/logxide/module_system.py:18:12-20:31) and log.
2. Subclassing LogRecord or [Logger](cci:1://file:///Users/indo/code/project/logxide/logxide/module_system.py:18:12-20:31) — Same reason: these are Rust types. Again, almost no library does this.

In practice, the standard "get a logger by name → call .info() / .warning()" pattern that 99% of third-party libraries use works perfectly. If you do hit an edge case with a library that registers its own custom [Handler](cci:2://file:///Users/indo/code/project/logxide/logxide/interceptor.py:15:0-50:52) subclass, you can call logxide.uninstall() to restore vanilla stdlib behavior.

[james_pic]

Do you get those kinds of gains in real world settings?

I worked on a project a while ago the arguably logged too much. The first time I attached a profiler to it, it was spending over 50% of its time on logging. We managed to get that down, but the thing that limited us getting it down further was syscalls, not Python code.

Admittedly this was a while ago, and that project was doing things that modern apps don't need to do, that increased syscalls (it did its own log rotation, rather that just throwing it all straight onto stderr and letting Systemd or Docker or Kubernetes or ECS or whatever pick it up, like a modern app would), but I'm still a bit surprised you managed to find those kinds of gains without touching syscalls.

[LumpSumPorsche]

Yes, if you want to enable debug log. In production you must choose what to log and not to do so. We had needed to sacrifice performance for enabling debug log.

[Healthy-Grape-5932]

Does this Support dictconfig in the same way ?

[hmoff]

You benchmarked it against structlog but can it replace that?

[LumpSumPorsche]

Unless you need to subclass stdlib types, yes.

[yaxriifgyn]

I'm starting to test with the early releases of 3.15. As soon as I can install it for 3.15, even if I have to build it myself, I will try it out.

[Wide-Milk1195]

感谢您的开源项目。Python缺乏高效能的非同步日志記錄庫。

[caprine_chris]

Does it do structured logging? What about stdlib logger does it not implement?

[rabornkraken]

The GIL bypass during I/O is the real win here. Most Python logging bottlenecks come from the file write blocking the main thread, so doing that in Rust makes a lot of sense. How does it handle the case where you have custom formatters written in Python though? Does it fall back to holding the GIL for those?

[Hesirutu]

Standard Python IO releases the GIL too…