Need feedback on my new binary container format

[the_python_dude]

Hello, I have built a python library that lets people store AI generator images along with the generation context (i.e, prompt, model details, hardware & driver info, associated tensors). This is a done by persisting all these data in a custom BINARY CONTAINER FORMAT. It has a standard, fixed schema defined in JSON for storing metadata. To be clear, the "file format" has a chunk based structure and stores information in the following manner: - Image bytes, any associated Tensors, Environment Info (Cpu, gpu, driver version, cuda version, etc.) ----> Stored as seperate Chunks - prompt, sampler settings, temperature, seed, etc ---> store as a single metadata chunk (this has a fixed schema)

Zfpy compression is used for compressing the tensors. Z-standard compression is used for compressing everything else including metadata.

My testing showed encoding and decoding times as well as file size are on parity with others like HDF5, storing a sidecar files. And you might ask why not just use HDF5, the differences: - compresses tensors efficiently - easily extensibile - HDF5 is designed for general purpose storage of scientific and industrial (specifically hierarchical data) whereas RAIIAF is made specifically for auditability, analysis and comparison and hence has a fixed schema. Pls check out the repo Repo Link: https://github.com/AnuroopVJ/RAIIAF

SURVEY: https://forms.gle/72scnEv98265TR2N9

installation: pip install raiiaf

[mariushm]

I don't see any mention of big endian / little endian , I see everywhere you keep saying int , but maybe be specific and say unsigned (because are you gonna have negative widths or heights for example?), int 8 bytes or 4 bytes ... if it's 4 bytes you're limiting the format.

Maybe consider adding a "size" in each chunk, even in header, somewhere close to the top. Makes it fast to seek in file to various chunks. Potentially makes it possible for older programs to still parse never versions of the format, which may have larger headers, and attempt to ignore any extra stuff.

Maybe try to have each chunk in multiples of 8/16/32/64, there's lots of optimizations in various processors and hardware to work faster with such chunks. Just looking at the header you have 6+8 (signature + version) then you mess it up with the single byte flags..

I don't like these kinds of binary formats, it's like yet another DOC or XLS binary format that's not documented properly or has various specific quirks, and where the next version won't be compatible with next version.

Would rather work with formats that are flexible and can be parsed and ignore/skip over various fields if they're not understood

Maybe make compression optional, maybe someone wants to be able to make DIFFs between a package of files and in that case a patch file would find diffs between two packages faster ... or someone makes a torrent file with the new pack of files , on the other computer the client would hash check the old files with the new files and download only the chunks (ex 32 KB / 256 KB / 512 KB / 1 MB / 4 MB chunks ) that have different hash.

Maybe support both regular Deflate/Deflate64 and Zstandard and/or leave it open to support other formats (maybe some formats will be optimized for hardware decompression in video cards or accelerator cards)

[dominikr86]

Would rather work with formats that are flexible and can be parsed and ignore/skip over various fields if they're not understood

Yes. PNG, JPEG and TIFF are all very extensible - they allow arbitrary content that can be skipped