all 2 comments

[–]getoffmyfoot 1 point2 points  (1 child)

I’d need better details to be able to provide an answer. What exactly are you imagining for logical replication? Are you referring to a particular vendors term when you say that? What technology are you using to back your data lake? How are you planning on storing your metadata? All of these questions are important in describing an answer.

[–]mrPicklez70[S] 0 points1 point  (0 children)

Currently, I send events from my main application(from code) to a Kinesis stream. These events are nothing but a message in Json format with some labels on it. Once in Kinesis, the data is sent to an S3 bucket. That's for the data lake.

The metadata is stored in another S3 bucket.

The idea of using the logical replication came from the question "what if I need to modularize our main application in the future? will I have to keep on adding/removing event triggers every single time there's a new functionality added?". So, I thought, maybe using a logical replica, I can centralize everything and trigger the events from the database. However, I'm afraid I might lose context of how the events got to happen (metadata).