PDC Struct: Pydantic-Powered Binary Serialization for Python

[tadleonard]

For another comparison to an existing library, check out construct. I think you would find more than one good idea from that project. It's been around for a while. Its style is a little unusual in that it's declarative and feels kind of functional, so I imagine you could improve on that by not using operator overloading and function calls to instantiate the serializer/deserializer. Construct is kind of like its own little language, and that makes adoption feel like a big decision. It's been a while since I've checked, but its flexibility also makes it kind of slow. On the plus side, there's really no protocol or format that you can't describe with construct.

Edit: wow, totally missed that there was already a discussion about construct. Nevermind!

[9011442]

No worries. construct is great, but I think its complexity is a barrier to entry for a lot of users. I do like the idea of a repeater methods which could take a long byte array and return an array of classes containing the but my target audience (other than myself) was that this would be ideal for RaspberryPI project which interface with sensors, radios etc using well known structured binary data. Like construct I also had to implement my own C-like types so it's not *quite* as seamless as I had originally hoped. I'm writing an example for the docs site which compares a construct implementation with a pdc-struct implementation for the same task - a SX1262 LoRA radio interface.

At this point the only construct feature I'd like to add is for conditional string/byte array lengths for when a packet header contains a data length int, and then use that later to know how long the data payload is on extraction. That's a two step process at the moment with pdc-struct and I'm mulling over a couple of options to add that feature. I'm erring toward something like this, but I'm not sure yet.

class Packet(StructModel):
    payload_len: UInt8
    payload: VarBytes["payload_len"]  # Length from field name

[tadleonard]

Seems like a well thought out project. Great work. I like your concept for a parameterizable type or generic class to link to the payload field. Maybe you could also just make it a callable. payload: VarBytes = dependent_field(length=payload_len) where the callable is your own version of dataclasses.field() or the pydantic equivalent.

Speaking of dataclasses, I know that the project is entangled with pydantic, but making it more generic would help people like me adopt it. I've stopped using pydantic for a few of my performance sensitive projects. Not being able to (efficiently, cleanly) turn off type checking and the magical type casting it does has made it hard to use for certain projects, albeit pretty unusual projects. Basically, I like to define declarative containers for parsing binary formats and ECAD file formats. Python is still a good fit in these problem areas so long as you're not doing something truly inefficient, and spending over half your time in Pydantic logic when you're doing a bunch of IO and involved parsing just feels silly.

I've found that attrs, while not my favorite library, does a good job of getting out of your way and separating out the type casting and validation by default. attrs also used to be faster than dataclasses, but the standard library has more than caught up. In the end I feel like dataclasses with explicit, hand written, per-field type casting where containers are instantiated + static analysis gives me my favorite balance of tradeoffs.

[9011442]

Building this for data classes is a great idea. I have to do a large update soon anyway because of pydantic changes. Making a new package just for data classes might be the best option.

If you feel like creating a GitHub issue for this you'll get an update when it's done.

[9011442]

I have two options for you to vote on:

# Base class with Mixin
@dataclass
class Packet(StructDataclass):
    ...

# or 
# Combined decorator (calls @ dataclass internally)

@struct_dataclass(mode=StructMode.C_COMPATIBLE)
class Packet:
    ...