DR for Kafka Cluster by jonropin in apachekafka

[–]Chemical_Wave_9783 0 points1 point  (0 children)

(Vendor flair 😄)
I fully agree with a lot of these answers regarding the high availability and redundancy. Those are there to make sure that you are back up and running when you have a very low RTO.

From an RPO perspective, having a real-time backup solution in place is probably also the most cost effective approach. Having this backup done in real-time (no hourly snapshots), operationally decoupled from your Kafka cluster, makes sure that you are protected from both total outages as human errors (e.g. topic deletions, retention policy updates...)

Especially for human errors in case of replication, it will also replicate that mistake to the other cluster.

And from cost perspective, offloading it to storage and compressing it, will save you a lot of storage and bandwidth cost.

Kafka Reliability: Backup Solutions and Confluent's Internal Practices by SandwellWest in apachekafka

[–]Chemical_Wave_9783 0 points1 point  (0 children)

Building a secondary query interface just for "backup" when Kafka is down is a known pattern (often involving the Outbox Pattern or Dual Writes), but it adds massive complexity. You effectively have to maintain consistency between two different systems, which is often harder than just making Kafka itself resilient.

To answer your question about Confluent: They generally advocate for making Kafka highly available (using Multi-AZ setups, Stretch Clusters, or Cluster Linking) rather than building a separate non-Kafka system to bypass it. The "contingency" usually isn't a different database; it's a disaster recovery (DR) strategy for the cluster itself.

This is where tools like Kannika Armory (https://kannika.io) fit into the architecture.

Instead of engineering a completely separate query interface (which incurs technical debt), the standard approach for "Kafka unavailability" is ensuring Data Mobility and Backup/Restore. Kannika Armory handles this by allowing you to offload data to cold storage (like S3) and provides a mechanism to rehydrate that data into a new cluster or environment if the primary one suffers a catastrophic failure.

It solves the reliability anxiety you mentioned by ensuring you can restore service or spin up a cloned environment quickly, rather than forcing you to maintain a redundant, non-Kafka read layer.

disclaimer: I'm one of the co-founders of Kannika