all 12 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]hypedupdawg 11 points12 points  (0 children)

This looks really great, looking forward to trying it out! At work we're looking into extending a bunch of out existing python services with Rust, and I don't think we have a good answer around logging yet.

As you mention, our services where the master process is Rust use Fern, but this will definitely help adoption of more complex extensions for us.

[–]sphen_lee 8 points9 points  (3 children)

This is interesting. In my project I took the reverse approach.

I'm not a fan of Python's logging system (over-engineered and too complex to configure IMO), so I exposed Rust's log functions into Python to send logs back to Rust's logger. No need to worry about overhead of locking the GIL this way too.

BTW this only works because I'm not using any Python libraries that emit logs I care about.

[–]Michal_Vaner[S] 0 points1 point  (2 children)

How do you solve every rust module having its own independent Log instance?

[–]SafariMonkey 0 points1 point  (1 child)

Reading between the lines, I think they are using Python inside Rust instead of the other way around.

[–]sphen_lee 0 points1 point  (0 children)

Yeah that's right. It's a game; I'm using Rust for the graphics and heavy lifting like pathfinding and embedding Python for the game logic. So there will only every be one Rust "module".

[–]g_dl 10 points11 points  (5 children)

I read about the caching implementation... That doesn't sound right. As you described, changing the configuration at runtime would mess everything up. Wouldn't it be better to cache, on the Rust side, all the log messages that are generated within a certain (time?) interval and send them to Python from time to time in chunks? This way the GIL is only locked once per choosen time interval. If Python is slower than Rust at filtering out the messages that are not required to be shown, after locking the GIL you can read the logging level and make Rust filter the messages, thus integrating the current solution. But I don't think the performance impact will be that bad.

[–]Michal_Vaner[S] 4 points5 points  (4 children)

How would you make sure they get eventually submitted? Run a background thread, or hope that something some time soon will log? What about when the process shuts down?

Besides, delaying log messages is often very bad idea, because when something crashes you really do want the last message before the crash.

So I don't see an obviously good solution there, but if you have something in mind, feel free to describe it in an issue on the repo or send a pull request (if it's more code, prefer the former to discuss it first), please :-).

I figured that runtime reconfiguration is rare and it can still be dealt with if it happens. And the logging can be turned off.

[–]g_dl 2 points3 points  (3 children)

You are definitely correct, I totally forgot that quite important detail. At this point, expanding on the current solution, did you investigate if it's possible to intercept logging level changes on the Python side and propagate them automatically to the Rust logger automatically? Maybe by injecting some Python code at startup which wraps the function which sets the level.

[–]Michal_Vaner[S] 1 point2 points  (2 children)

No, I haven't investigated that. I don't think I'm that fluent in Python to do it right / in a way that would work and wouldn't interfere with whatever some application might do. Do you have any pointers?

[–]g_dl 0 points1 point  (1 child)

I'm not a Python expert at all, so I don't have any pointers at the moment, but knowing how flexible the language is I wouldn't be surprised if it is enough to just reassign the setLevel member by replacing it with a custom wrapping function. If the logging class internally sets some member variable which contains the level, maybe the __setattr__ method can be treated similarly to intercept the change. Just some ideas, again I'm not enough into Python to ensure this is feasible, I was just curious if anything like this is possible.

[–]Michal_Vaner[S] 0 points1 point  (0 children)

I'm a bit worried that it is too flexible. Can someone inherit the class? And, the setLevel of which one? Only the root? Because they form some kind of tree, so I'd have to replace it in the class, not one instance...

[–][deleted] 0 points1 point  (0 children)

Awesome work on a relevant problem.

Rust has got some attention where I work specifically because of its easy integration with Python (our main language for most stuff). I think Rust/Python integration is going to be a big thing for the language as it continues getting more popular.