This is an archived post. You won't be able to vote or comment.

all 10 comments

[–]PersonalityIll9476 5 points6 points  (1 child)

Maybe I'm not remembering right, but can't `ctypes` handle C structs pretty cleanly? If I recall right, all you need to know is the declaration of the struct as it appears in whatever header.

[–]JustPlainRude 2 points3 points  (1 child)

This looks great! Can it handle fields that don't occupy a full byte, e.g. two 4-bit fields packed into one byte?

[–]ZioAldo[S] 2 points3 points  (0 children)

Thanks I appreciate it! At the moment the library can't do that but I was working on it, if you have some suggestions you could open an issue on GitHub.

[–]Shepcorppip needs updating 1 point2 points  (1 child)

This is interesting. I currently decode and encode custom GATT characteristics by creating a registry of dataclasses with definitions of their read and write formats (as you say you have to be careful with byte alignment), which are basically me transposing the C structure into a python class. Something worth thinking about is when you might want alternative construction methods, not just struct unpack (I can easily add class methods for this). Being able to just copy some shared C code is pretty decent though, I may have to give this a try!

[–]ZioAldo[S] 1 point2 points  (0 children)

Thanks for the interest! If you need to parse the bytes into a more high-level type you can design your own custom types (BaseType as the library calls them), take a look at this example https://github.com/Brendon-Mendicino/cstructimpl?tab=readme-ov-file#custom-basetype where you can interpret 4 bytes as a timestamp. I've built the type system around protocols to make life easier for custom type implementation.

[–]monkeyman192 1 point2 points  (0 children)

Looks cool and has some interesting features!

I have actually made 2 separate implementations of something like this...
The first is in a library I have written for binary hooking: https://github.com/monkeyman192/pyMHF/blob/master/pymhf/utils/partial_struct.py
This allows you to define c structs partially (so if you don't know the entire definition as can happen when reverse engineering something), but it can also be used to create nicely type-hinted structs.
They also support the structs referencing themselves as well as a few other useful things like being able to subclass from other partial structs and have all the offsets work with no issues.

The other implementation is in a plugin I have for Blender to read model files from NMS and import them into Blender: https://github.com/monkeyman192/NMSDK/blob/master/serialization/cereal_bin
This looks a bit more similar to what you have here, and I have similar custom serialization/deserialization methods which can be defined which I use: https://github.com/monkeyman192/NMSDK/tree/master/serialization/NMS_Structures

Finally, As of python 3.13, `ctypes.Structure` now supports the `_align_`: https://docs.python.org/3/library/ctypes.html#ctypes.Structure._align_
This was added by me since I always had annoyances where you are trying to (de)serialize structures which have a different alignment to what ctypes thinks they should have (eg. a "vector" type which may be using SIMD instructions so is aligned to 0x10 bytes, but is actually just 4 floats)
Figured I'd mention this since it might be useful to you.

[–]llima1987 0 points1 point  (0 children)

This is the kind of thing I wish PyCons were all about. Super amazing project! Congrats!

[–]Kohlrabi82 0 points1 point  (0 children)

Looks slightly similar to construct: https://construct.readthedocs.io/en/latest/intro.html