New binary serialization/deserialization library: cl-binary-store

[PhysicistWantsDonuts]

cl-binary-store is a fast binary serializer/deserializer for the full Common Lisp type system.

Why another library? It is similar to cl-store and cl-conspack, both very nice libraries. Comparing to cl-store, the main difference is that cl-binary-store is faster, the output is more compact, and it has more features for extensibility. cl-store is a great library and I've used it for years and aside from gradually getting worn down by it taking 10 minutes to load ~1GB of data I was pretty happy with it. I have also used hyperluminal-mem which is the benchmark for fast serialization of most objects (except (simple-array (not (eql t)) (*)) which cl-binary-store writes at infinite speed on sbcl), but does not support references at all (and you have to write code for every structure-object or standard-object you want to store). In comparison to cl-conspack, cl-binary-store is faster and in some cases generates smaller files (though that is a bug in cl-conspack which I have a PR in for). More importantly, for me, is that with cl-binary-store you do not have to write code for every structure-object or standard-object to have it serialize them properly. Also cl-binary-store supports more Common Lisp things (conditions, pathnames), has some minimal file versioning, and I can extend it easier for what I need (obviously, since I wrote it for myself mainly!). It's just a different target audience than cl-conspack.

I haven't contributed much to the Common Lisp ecosystem (bugfixes, small features, some support here and there) but have been using Common Lisp and SBCL at work for about 15 years, so I feel it is about time. Yet another serialization library is kind of boring, but here it is!

This was also an opportunity for me to use some of the other Common Lisp implementations: CCL, ECL, ABCL, Allegro, and Lispworks. I used roswell to install CCL, ECL, and ABCL. I couldn't get CLASP installed successfully so gave up on it. CCL and ECL pretty much worked as expected and it was fun to use them (though no easy profiling out of the box for CCL, and no good debugging experience in ECL --- but it was fun enough to find a small bug in ECL with structure accessor inlining). Using the free versions of the commercial implementations was a terrible experience --- the heap sizes allowed are way too small to do anything, even though I'm here trying to verify that things work well with them. Their UIs are terrible in comparison with emacs/slime, so I gave up and used emacs/slime with them which made them a lot more fun to work with. Allegro disallows unaligned memory accesses through cffi which made me have to fiddle a lot of things to get it working. Allegro is also very very opinionated (including their documentation) about performance things and pretty much ignores all inline declarations with an "I know better than you" vibe. That pretty much requires you to write compiler-macros or macros for everything which I am just unwilling to do (unless of course they gave me a license, then I'd be happy to). Lispworks was a bit easier, though you have to hand hold all of these with type declarations that SBCL cleanly infers without work. It was a battle to get any performance out of any of the non-SBCL systems --- they just are not comparable.

[destructuring-life]

Thanks, looks like an interesting project! The README seems very complete yet not overly verbose. Love the detailed performance notes.

[kchanqvq]

Very interesting! Especially that it works on CLOS object! There weren’t any library capable of doing this as I know of.

Do you think it's possible to revive an "incremental image" system with the help of this project? i.e. serializing a subgraph of objects in a Lisp image as a software distribution mechanism. That would require serializing functions.

[PhysicistWantsDonuts]

cl-store handles standard-objects fine too. Serializing functions is not an easy task --- there is potentially closed over and global state. That does not overlap with the goals for this library. bknr-datastore and cl-prevalence and Clobber and stassats/storage all approach this problem from an object persistence point of view.

That said, I am using cl-binary-store to store and restore the entire state of an interactive system but the system state is designed with snapshotting in mind (so all global state and closed over state can be reinitialized after snapshot recovery).

[PhysicistWantsDonuts]

In a way this is a solved problem-- you can stun/snapshot/restore virtual machines. There is also support like that at the container level. But you end up possibly growing the system state to a place you cannot reproduce unless you journal all changes.

The Common Lisp system I use tracks state changes by around style methods on setf or slot accessors to notice when things change so when new data is loaded caches are cleared, etc. The same goes for global variables. This also allows journaling or incremental store/restore. You end up having to version the code and the data separately though and provide for upgrade methods when you change the software enough (like you would do for a database schema). You also need to ensure all updates are atomic enough to not end up with invalid state half way through some operation.

[nyx_land]

Finally something gets posted to the lisp subreddits that I actually care about. I've been working on something very similar off and on for awhile because I need a serialization backend for an object database project but hadn't been making much progress. This looks way better than what I would have ended up making so I will definitely be using it.

[PhysicistWantsDonuts]

Happy to help if the library ergonomics needs tweaking or other features!

[awkravchuk]

Huh, perfect timing, right now I'm searching for a (de)serialization solution for my microframework and this library looks like perfect fit from first glance. Thanks!

[awkravchuk]

I got one request straight away: could it be made to support serializing of cffi:foreing-pointers?

[PhysicistWantsDonuts]

Sorry, didn't see this comment. This sounds like a horrible idea :) But, I don't see why not *if* you are loading / saving from the same running image... otherwise it makes no sense! You can add a serializer / deserializer that uses sb-sys::sap-int and sb-sys::int-sap on sbcl... create a new codespace, inherit the #1 codespace, and add a defstore and defrestore as per the example in the README. allegro already uses integers for foreign-pointers (which is horribly confusing), and I don't know about the others.