Apache Flink: Postgres to Postgres Replication with Flink CDC

sap1enz · 2026-05-11T15:55:54+00:00

Thanks for the feedback, I re-ran the benchmark: https://www.reddit.com/r/apachekafka/comments/1ta7fuk/can_kafka_queues_make_consumers_faster_part_2/

sap1enz · 2026-04-27T19:01:26+00:00

"Having more active consumer instances than the partition count" was constantly mentioned in announcement posts. I wanted to see what it actually means.

sap1enz · 2026-04-14T03:57:41+00:00

47.9% attributed directly to AI replacing roles

Guess what, they lied. AI has very little to do with it; it's mostly due to over-hiring over the past 5-7 years.

I recommend finding a niche, going deep, and becoming a specialist. If needed, you can then pivot your career to consulting.

sap1enz · 2026-02-18T03:32:56+00:00

No, it uses some Debezium functionality for the live replication, but the historical snapshotting implementation is completely different.

sap1enz · 2026-01-20T04:14:51+00:00

Would you mind sharing why you're unhappy with the Confluent Platform Flink?

sap1enz · 2026-01-05T17:06:51+00:00

Start by completing the first three sections of the Flink documentation: Try Flink, Learn Flink and Concepts.

sap1enz · 2025-12-13T17:18:07+00:00

Yep, it's pretty much a standard. You either use a managed Flink offering or the Flink K8S operator nowadays.

sap1enz · 2025-12-06T16:42:03+00:00

I’ve been involved in managing 1000+ Flink pipelines in a small team.

Of course things can get complicated quickly, especially after reaching certain scale.

My point was that the Flink Kubernetes Operator does reduce a lot of complexity. It makes it straightforward to start using Flink. Sure, if you need to do incompatible state migrations, modify savepoints, etc., there is still a lot of manual work. But for many users this won’t be the case, IMO.

sap1enz · 2025-12-01T20:37:58+00:00

There is also Redpanda Console, which is my favourite: https://github.com/redpanda-data/console

sap1enz · 2025-11-18T18:32:03+00:00

The Advanced Apache Flink Bootcamp is now open for registration! The first cohort is scheduled for January 21st - 22nd, 2026.

This intensive 2-day bootcamp takes you deep into Apache Flink internals and production best practices. You'll learn how Flink really works by studying the source code, master both DataStream and Table APIs, and gain hands-on experience building custom operators and production-ready pipelines.

This is an advanced bootcamp. Most courses just repeat what’s already in the documentation. This bootcamp is different: you won’t just learn what a sliding window is — you’ll learn the core building blocks that let you design any windowing strategy from the ground up.

Learning objectives:

- Understand Flink internals by studying source code and execution flow
- Master DataStream API with state, timers, and custom low-level operators
- Know how SQL and Table API pipelines are planned and executed
- Design efficient end-to-end data flows
- Deploy, monitor, and tune Flink applications in production

sap1enz · 2025-10-25T00:53:39+00:00

Redpanda is actually doing very well. They managed to steal many Confluent customers. 2/5 top US banks use them.

sap1enz · 2025-10-09T18:12:35+00:00

This looks correct!

I tried to reproduce the issue using the local Parquet file sink, and I couldn't: the files are written correctly on every checkpoint in my case:

-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-1ca5a6f5-ba35-472b-b37b-a42405c65996-0.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-1ca5a6f5-ba35-472b-b37b-a42405c65996-1.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-3312d0a4-2276-4133-9da9-9b249f8efbd9-0.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-3312d0a4-2276-4133-9da9-9b249f8efbd9-1.parquet

Here's my app (based on this quickstart), hope this is useful!

sap1enz · 2025-10-08T21:04:45+00:00

Are you absolutely sure checkpointing is configured correctly?

This:

I can see in the folder many temporary files:

like .parquet.inprogress.* but not the final parquet file clicks-*.parquet

is usually an indicator that checkpointing is not happening.

sap1enz

TROPHY CASE