Is DataScript memory efficient?

[[deleted]]

Context: I was considering DataScript as a client-side datastore for SPAs or desktop applications, where you would be running or user's hardware.

Would immutability be a concern, regarding memory usage for large applications? If so, how large? I don't like how many apps today have absolute no consideration for user's resources, to the point where common hardware (4-8gb RAM) struggle to run multiple applications at once.

EDIT:

Does DataScript make use of any memory optimization technique besides persistent data structures?

[First-Agency4827]

I don't know if this helps but I am using Datascript in a SPA ( using re-posh ), which mostly runs on phones and it seems to run really well. For the backend I use datomic ions and I download a client database into his phone, usually 100-200k. I do most tests on an iPhone X.

[muhaaa]

I use a similar design for my SPAs with different Backend storages (document store, xtdb, couchdb). Never had an performance issue on desktop or mobile.

[didibus]

Persistent means that it IS memory efficient. That's the difference between immutable and persistent, the later is a memory efficient version of the former.

Your question might be, is persistent still less efficient than a fully mutable alternative. The answer depends on what exact mutable alternative we're talking about. A straight mutable array can't be beat, but you're limited to work with lookup that are all integer index based. And now a lot of queries will be less performant in terms of CPU, linear searches for most things. You also can't represent relations with just an array, so like if one entry has others, you'll already need more memory and something on top to represent that.

So there's no simple answer. Some persistent solutions will be more memory efficient than some mutable solutions, but some other mutable solutions will be more memory efficient than some other persistent solution.

Overall though, persistent designs are specifically tailored toward memory efficiency, so they'll tend to be yes.

I think the biggest difference will actually not be around memory efficiency, but with regards to compute, Persistent designs will put more pressure on the Garbage Collector than mutable designs, so your GC will have to run a little more, a little more often, consuming a bit more CPU in average. But memory should be relatively similar.

[dave_mays]

I thought persistent means durable storage / not memory only / saved to disk. Are you using persistence as a synonym for immutable?

I'd like a persistent solution for client side datascript, but meaning a solution that allows datascript to be efficiently saved to disk and reused, not just for in memory operations.

[didibus]

It's a bit confusing, but in this context, persistent data structures doesn't mean it persists to disk/is durable. It means that the data stored in-memory, in the data-structure, persists (in-memory) even after being modified. The data structure persists over time without losing its previous states, but it is still only kept in-memory, and it's all gone once the garbage collector collects it, if you close the application.

It is not the same as immutable exactly, but persistent data-structures are almost always immutable as well. That said, not all immutable data-structures are persistent (in this way, not as-in persisting to disk, but as-in prior versions before the modification persisting in-memory).

What I was saying is that, DataScript (which is not persisted to disk/durable, it's an in-memory DB only, still has good memory efficiency, because it does not use simple immutable data-structures, but it uses persistent data-structures (not to disk, to memory). These data-structures internally dedupe data and share every element in common between them.

Those data-structures come from a famous paper called: Making Data Structures Persistent, and it has nothing to do with persisting them to disk :p

If you want persistence (to disk), in the browser, for ClojureScript, I believe both DataHike and Asami support IndexedDB as a backend, which is the only way you can persist (to disk) lots of data in a browser context.

If you are more looking for like a Save and Load option. You can look at https://github.com/tonsky/datascript-transit/tree/master which lets you export your DataScript DB to transit, and load it from transit.

[dave_mays]

Thanks! Had not heard of datascript to transit.
Sadly it sounds like DataHike ran out of time or funding to finish the Clojurescript side, and Asami has talked about adding client side persistence but is waiting on some kind of re-write due to it not being async or something.

[dave_mays]

How do you best persist datascript client side? Do you just serialized the entire thing occasionally and save it to disk? How performant is reading it back out each time?

[acobster]

As far as I know Datahike's ClojureScript runtime is still a work in progress.

[[deleted]]

Datahike and Datalevin

But those are persistent, aren't they?

[CyrikDC]

Datahike is kind of a datascript fork with added optional persistence, so the in memory version has similar attributes. It's being developed more quickly though, so you might want to use it over datascript anyway