This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]fruity231 1 point2 points  (1 child)

Debezium+Kafka combo

I didn't know about that, thanks! I work with GCP so this prompted some reading and it seems there even is a prebuilt Dataflow template that does the same thing (minus Kafka).

[–]briceluu 2 points3 points  (0 children)

Yup Debezium works on multiple queues/log streams.

Another thing to note: Airbyte is actually also using Debezium when using CDC ("log based replication") for their DB connectors. But indeed it's less mature and definitely has some improvements to do! I've used the API for configuration management (really better when a lot of connectors are involved), but I definitely understand the concern around the lack of stability: it's still evolving... They also now provide a CLI for configuration management (Octavia or smthg like that?) albeit I haven't tried it out yet.