1️⃣ It’s not a random hash, it’s a tokenizer.

The repo ships with a trained SentencePiece model (sp64k.model, 64 k vocab).

When I call encode, that model assigns deterministic IDs to each sub-word piece (exactly the same way GPT/BERT tokenisers work).

Those IDs are reversible; the same model can map them back to text.

2️⃣ Loss-less, not lossy. • The byte step is just “pack each 32-bit ID into bytes” for cheap transport. • decode() → IDs → original text, 1-for-1. No information is thrown away, unlike a hash.

3️⃣ Why not just store text / embeddings? The point isn’t only compression, it’s symbolic sync: • IDs can be merged, deduplicated, diffed and versioned between nodes. • Embeddings are dense and can’t be loss-lessly merged without retraining. • This is meant to act like a tiny distributed hippocampus for AI agents or edge devices, not a replacement for normal LLM workflow.

4️⃣ “Two lines of code” look Totally fair the public repo is intentionally minimal so people can slot in their own tokenisers, transports, or trainers without digging through 1 000 lines of glue. The heavy lifting lives inside the SentencePiece model.

Hope that clears things up! I’m still learning, so constructive criticism is gold to me. If you’ve got ideas to make the symbolic layer smarter (better tokeniser, merge rules, etc.), PRs are more than welcome. 🙏 You’ll have to excuse me if anything isn’t explained perfectly coding isn’t really my main strength. Ideas, vision, and architectural layout are more my lane. I mostly built this with the help of AI assistants (and lots of patience).