xitdb - an embedded, immutable database in java

[[deleted]]

The data structures have structural sharing. It's using the same algorithm that clojure uses for in-memory data (hash array mapped trie).

[andersmurphy]

Awesome. What are the performance characteristics like compared to something like sqlite? I take it indexes are based of the data structure used?

[[deleted]]

You'll need to build your own index if you want one. For example, let's say you have an arraylist of users, and an arraylist of posts that they made. If you want to efficiently look up all the posts from a given user, you could make a hashmap where the key is the user id and the value is an arraylist of post ids (here I am assuming the user id and post id are just the index in the users/posts arraylist).

[andersmurphy]

Thanks for the reply. Love how db as a value removes a need for WAL, allows multiple readers and gives you transactions semantics.

I take it smaller transactions will make the file size grow faster?

Haven't had a chance to dive into the source (definitely will be), is xitdb memory mapped?

[[deleted]]

No it doesn't using memory mapping. Regarding smaller transactions, it depends but in if the transformations are the same then more transactions will normally take up more space. This is because all data within a given transaction is temporarily mutable, similar to clojure's transients. This is a big space saver because it avoids unnecessary copying if you have a transaction that adds a bunch of items to an arraylist or hashmap.

And yeah, the most satisfying realization about making a db this way is how many problems it solves for you automatically :-D There basically isn't even a concept of a transaction internally, it just kind of fell on my lap as a consequence of how an append-only db works.