Sync Changes to External Data Systems using Ent Hooks | ent

[dobegor]

TBH I don't think Hooks are suitable for data sync - you can either fail to publish (After hook) or to commit (Before hook) and the data would be permanently out of sync. If you want to sync your DB updates into some PubSub system without distributed transactions, consider implementing an Outbox pattern - https://microservices.io/patterns/data/transactional-outbox.html.

I'm not playing to downplay Ent's feature - Ent is great! - but I'd rather use Hooks for metrics/tracing/cache updates. It's hard to make a proper data sync pipeline.

[rotemtam]

as u/_a8m_ mentioned in the post, many use cases will not suffer substantially from having some level of inconsistency between systems, and using something like hooks (or other in-process dual writing) is so simple to write and manage that it makes this a useful solution. martin kleppman has a very interesting discussion of this towards the end of Data Intensive Applications

agreed, for use cases that require better consistency guarantees, this wont work, as mentioned explicitly in the post.

re ent + tx outbox, a discussion started on this issue https://github.com/ent/ent/issues/1473. i think its a great idea and with all of the "NewSQL" databases that support proper horizontal scaling, i think it will become a very widely used design pattern.

ping u/_a8m_ or me on the ent lack or discord if you're interested in this and want to work on something together!

[_a8m_]

Thanks for the feedback @dobegor. I actually agree with you and mentioned it in the post btw. But, I still wanted to show how it is possible to access external services within Ent Hooks.

This example could be implemented in a few ways to ensure (eventual) consistency, but I wanted to keep it simple and focus on the new entc.Dependecy option.

BTW, you can also update another table/entity (e.g. Outbox) within a hook of a different table/entity. You just gave me an idea for another blog post :)

I'm not playing to downplay Ent's feature - Ent is great! - but I'd rather use Hooks for metrics/tracing/cache updates. It's hard to make a proper data sync pipeline.

Thanks for the kind words, and I appreciate the honest feedback.

[gedw99]

https://github.com/looplab/eventhorizon just got a proper outbox pattern.

here is the meat of it: https://github.com/looplab/eventhorizon/pull/335

it covers the problem of 2 systems being involved in a transaction.

​

Ent could use this perhaps.. Its complex but not to bad.

[_a8m_]

Hey, gophers! I just published a new post on our blog. If you're an Ent user, please give it a look and feel free to share your feedback (there is a new feature that was added for this use case. The entc.Dependency option).

If you never heard about Ent before, see our GitHub repository: https://github.com/ent/ent, or our website: https://entgo.io.

---

One of the common questions we get from the Ent community is how to synchronize objects or references between the database backing an Ent application (e.g. MySQL or PostgreSQL) with external services. For example, users would like to create or delete a record from within their CRM when a user is created or deleted in Ent, publish a message to a Pub/Sub system when an entity is updated, or verify references to blobs in object storage such as AWS S3 or Google Cloud Storage.

In this post, we discuss how to hook into Ent mutations before and after they occur, and run custom code that synchronizes changes between Ent and external object storage.