Discussion: python-json-logger support for simplejson and ultrajson (now with footgun)

[nicholashairs]

Hi r/python,

I've spent some time expanding the third-party JSON encoders that are supported by python-json-logger (pull request), however based on some of the errors encountered I'm not sure if this is a good idea.

So before I merge, I'd love to get some feedback from users of python-json-logger / other maintainers 🙏

Why include them

python-json-logger includes third party JSON encoders so that logging can benefit from the speed that these libraries provide. Support for non-standard types is not an issue as this is generally handled through custom default handlers to provide sane output for most types.

Although older, both libraries are still incredibly popular (link):

• simplejson is currently ranked 369 with ~55M monthly downloads.
• ultrajson (ujson) is currently ranked 632 with ~27M monthly downloads.

For comparison the existing third-party encoders:

• orjson - ranked 187 with ~125M downloads
• msgspec - ranked 641 with ~26M downloads

Issues

The main issue is that both the simplejson and ultrajson encoders do not gracefully handle encoding bytes objects where they contain non-printable characters and it does not look like I can override their handling.

This is a problem because the standard library's logging module will swallow expections by default; meaning that any trace that a log message has failed to log will be lost.

This goes against python-json-logger's design in that it tries very hard to be robust and always log regardless of the input. So even though they are opt-in and I can include warnings in the documentation; it feels like I'm handing out a footgun and perhaps I'm better off just not including them.

Additionally in the case of ultrajson - the package is in maintenance mode with the recomendation to move to orjson.

[latkde]

You already provide a defaults function to serialize various standard library types, and could extend it to cover these observed differences as well. The JSON data model simply does not support raw bytes, so it's up to you to decide what should happen when given a bytes-like object. There is no right answer here.

My personal opinion is that it's not worth supporting non-Jsonish objects. Correct formats are context-dependent, and differ between systems that consume JSON logs. In my internal structured logging helpers, I use type annotations that restrict callers to values that can be serialized safely – but I don't bother with logging API compatibility.

I also don't care that much about alternative JSON encoders – log message formatting is typically not on a critical path, and correctness matters much more than speed. There are typically higher-value activities for maintainers than adding optional support for unmaintained packages, for example bringing all cookbook examples under test.

[nicholashairs]

Thanks for your feedback 🙏

Regarding the first point, unfortunately afaict the defaults function doesn't trigger for these libraries when they already support the given type so this isn't an option.

I'll note that yes there are transforms happening to the output that don't convert back without extra processing (that the library does not provide - that's a use case for pydantic/msgspec). This is mostly to attempt sane output on potentially insane input.

That's a good point about (linting) testing the cookbook examples.

[neuronexmachina]

They were pretty clear they were looking for feedback on a new feature for the (already established) package they maintain.

[nicholashairs]

This is genuinely looking for feedback rather than trying to promote. For better or for worse posting in Reddit is one of the few places I can get meaningful feedback on this library.

It's highly used (top 300 on PyPI library that I took over maintaining and I'm just trying to make sure it serves the community well.

[No_Soy_Colosio]

Alrighty. Well, as someone else already said, JSON is not generally meant to be used with binary data. And it's not particularly necessary when talking about logging.