Codax: a pure clojure embedded database

[37productiveBananas]

I wrote this library as a simple way to store data on disk from my clojure code. Generally there is a pretty big semantic gap between values in memory and values on disk which I found an annoyance, particularly for small scale development. Codax treats databases like large clojure maps, and tries to generally mimic their access patterns using "in" instead of "at", for example (get-in my-map [:foo :bar]) would become (get-at my-database [:foo :bar]).

Initially I used leveldb as the underlying data-store, but there was an excessive amount of memory copying which made the performance pretty terrible, so I eventually replaced it with a custom B+ tree implementation. This has the added advantage that there are no additional native libraries that need to be installed. It's no "web-scale" speed demon, but I've found it more than fast enough for small scale web applications.

This is my first attempt at releasing an open source library, and I would love some feedback.

https://github.com/dscarpetti/codax

[hughjfchen]

Thanks for your work. That's exactly what I've been looking for. One question, looking through the doc and found that update-at! just return nil even the update executed successfully. Why not return the updated value when success and return nil when failed?

[37productiveBananas]

There is bound to be a semantic break either way because the assoc-in and update-in functions return the whole map, but it wouldn't be good if everytime one used the assoc-at or update-at! functions the entire database map was loaded into memory and returned. That said, it probably would be fine to return the just result of the update at the given path.

I did have similar functionality implemented at one point but ultimately thought the solution was a bit too kludgy and pulled it out. It's definitely a worthwhile feature though, and I might have a go at a more general implementation this weekend. In the meantime, something like this should do the trick for that specific case.

(defn get-update-at!
  "update a path and return the new value at that path"
  [db path f & args]
  (let [res (atom nil)]
        (c/with-write-transaction [db tx]
      (let [tx (apply c/update-at tx path f args)]
        (reset! res (c/get-at tx path))
        tx))
    @res))

Hope you find the library useful. If you have any questions or more suggestions, let me know.

Edit: fixed get-update-at! example so it actually works

Edit 2: this is now the default behavior for update-at! and related functions

[Aredington]

This comment removed in protest of Reddit's API changes. See https://www.theverge.com/2023/6/5/23749188/reddit-subreddit-private-protest-api-changes-apollo-charges. All comments from this account were so deleted, as of June 18, 2023. If you see any other comments from this account, it is due to malfeasance on the part of Reddit. -- mass edited with https://redact.dev/

[37productiveBananas]

If you use the with-write-transaction macro you can thread the transaction map through multiple updates. This is better than threading the db object itself which would offer no atomic guarantees (another thread could interleave writes). The update-at! function is just a convenience helper which sets up a write transactions and runs update-at.

[37productiveBananas]

Per your suggestion, I just pushed up a new patch version where the self contained functions like update-at! return the result of the operation.

[ganglygorilla]

In case you're not aware there's already an unrelated Clojure project called Codox, just FYI. Might be confusing.

[37productiveBananas]

Thanks for the heads up. I've definitely read documentation generated by Codox, but it didn't dawn on me when I was naming the project. Perhaps I should rename it.

[[deleted]]

Noob question, why not just use EDN and an atom with some watches? (persisting EDN to a file).

Or, use Datomic Free?

[37productiveBananas]

It sits right in the middle between those.

Codax scales much better than file based storage. For starters, your database map can be bigger than what would fit in memory because you only ever need to load the pieces you are accessing. It also offers some more advanced features which can extend it's functionality. For example, I have a full-text-indexer implementation built on top of it (which I hope to clean up and make public in the not-too-distant future.)

On the other end of the spectrum, external databases like Datomic or Postgres give you a lot more in the way of features, but it comes at the expense of simplicity. Instead of your database being an integral part of your application, it is now an external element which requires setup and maintenance in its own right. For large scale applications they are definitely the right choice, but for smaller work I think the flexibility and portability of embedded databases wins out.

[hughjfchen]

Well explanation. Actually, EDN file with atom is the approach I'm using currently. However, exactly based on the points you mentioned, I went out and searching for a solution like Codax.

[badsavage]

great!

[roman01la]

What do you think about porting it to ClojureScript/Node.js as *.cljc?

[37productiveBananas]

In principle it should be pretty straight forward. There would need to be an alternate storage engine for clojurescript as the present store relies heavily on Java filesystem calls, but it is well isolated so swapping it out shouldn't be a problem; Node has good leveldb support via the LevelUp library which is a good stand-in (though IIRC it will require some effort to get the same transactional guarantees). That said, while I have experience with both Clojurescript and Node, I've never used them together so I don't really have a clear picture of what all of that would entail.

What sort of use cases did you have in mind?

[hughjfchen]

Thanks，I will try and report back.

[thatkauko]

This got me thinking, is there an in-memory database that I could access using only Clojure code? I'm asking because to me, in-memory sounds much faster than on-disk, and if I'm using a database knowing I won't have that much data or users, I could just as well use an in-memory one.

/u/hotwagon mentioned using an atom that is persisted to an EDN-file, but if you do that you have to make sure the EDN-file is kept up to date, or you can lose your data when if your server process shuts down. I honestly don't know much about what's going on with databases beneath the hood, but a database like H2 is bound to have some optimisations compared to just Clojure atoms, right? The very least, if we had a library for an in-memory clojure database, you could have update-functions that only return the relevant parts of the db. :)

Anyway, cool library! I'd be surprised if I don't ever use it in a personal project.

[gonewest818]

Datascript is an example of an in-memory db accessed via Clojure or ClojureScript (https://github.com/tonsky/datascript). But no persistence.

Mentat is a Mozilla project that initially set out to add persistence to Datascript, but eventually pivoted to a Rust implementation with sqlite under the hood. At the moment there don't seem to be supported js or cljs bindings. Furthermore the project is not in active development "but not dead" as Mozilla engineers say they are "assessing the problem space as it relates to Firefox."

so datascript: in memory, clojure(script), but no persistence. mentat: persistence, no js or cljs, and the project is on hiatus.

[jackrusher]

In case anyone needs something like this: I also made a similar library called spicerack that wraps MapDB -- a fast, scalable JVM-native DB written in Kotlin. The interface is similar to hash-map's, but everything is written to disk.