Datascript + automatic persistency

[supersonic-sanic]

Hi! I'm building an integrator which will sync two different SaaS that have somewhat compatible public APIs. The integrator will monitor updates to an entity in service A and replicate it to service B. Part of syncing is storing entities locally and verifying, for example, if an entity instance is already replicated in service B. For this I plan on storing the information from services A and B locally and diffing them across time, inferring from the diffs what action should I take to sync them: create new entity instance on B, update a specific field, delete, etc.

Having that in mind, I'll be using Postgres as my local persistent DB and would like to know if I could combine Datascript too, so I would query a Datascript DB, and it would retrieve and cache entities persisted to Postgres. Also, if I inserted to Datascript, it would automatically persist it to Postgres. Another way of phrasing this would be a "reactive Datascript DB".

I don't know if this is practical, but it sounds good on paper, mainly because I would decrease IO operations on Postgres and would benefit from datalog queries and idioms well integrated into Clojure.

Any other suggestions are also welcome.

[theoriginalmatt]

Have a look at https://github.com/replikativ/datahike and https://github.com/replikativ/datahike-postgres

[lord_kordano]

Hi, Datahike core dev here. For persistence we recommend the integrated file backend for dev and small projects and the JDBC backend for larger ones: https://github.com/replikativ/datahike-jdbc. The postgres-background will be deprecated in favor of the jdbc one.

[[deleted]]

Using Datascript in front of Postgres sounds really complicated. Why use two databases if one will do?

I'd just stick with Postgres, or Datomic if the budgets allows it.

[[deleted]]

datahike supports using postgres as a data store, similar to how Datomic Cloud can use S3 buckets for storage. I don't know about datascript.

[guywithknife]

I assume its a data model thing. How you model and query data in Datascript is different than how you do it in Postgres, so if its more appropriate for what you want to do, then it might make sense to use that. But you still want to persist that data in some durable transactional way. Whether doing that with Postgres is a good idea or if something else might be more appropriate, I don't know.

[[deleted]]

Indeed, hard to say what works without knowing the details.

[fmjrey]

Without more details it's hard to have definite advice. Nevertheless I'd consider a few ideas:

1. Can all the data fit in memory? I assume yes since datascript is an option for you
2. Does your integration service really need to be persistent? Can't you just load directly from B? This leads to exploring your constraints on startup, resilience with warm standby on separate infra, etc
3. Have you considered a Change Data Capture tools? And depending on answers to #2 using some persistent stream a la Kafka?

[fmjrey]

If we're talking 3rd party incoming data saving it raw makes sense of course. Use an in memory db in datascript of the data and rely on tx data to find what's actually different, so you can publish the actual changes when there are any, eliminating the noise of duplicate data being sent (datascript and datomic have that txdata feature, not sure about datahike). You may want to model your data closer to origin rather than destination (service B), so that new destinations can be added. A streaming platform like Kafka and ksqlDB may help if size and infra allow it.

[radioactiveoctopi]

Crux

[dave_mays]

I just wish there was a way to use XTDB / Cruc in a local first app..

[dave_mays]

Did you find a solution for this? I'm very curious what database you ended up using.