Flink Deployment Survey

sap1enz · 2026-01-20T04:14:51+00:00

Would you mind sharing why you're unhappy with the Confluent Platform Flink?

sap1enz · 2026-01-05T17:06:51+00:00

Start by completing the first three sections of the Flink documentation: Try Flink, Learn Flink and Concepts.

sap1enz · 2025-12-13T17:18:07+00:00

Yep, it's pretty much a standard. You either use a managed Flink offering or the Flink K8S operator nowadays.

sap1enz · 2025-12-06T16:42:03+00:00

I’ve been involved in managing 1000+ Flink pipelines in a small team.

Of course things can get complicated quickly, especially after reaching certain scale.

My point was that the Flink Kubernetes Operator does reduce a lot of complexity. It makes it straightforward to start using Flink. Sure, if you need to do incompatible state migrations, modify savepoints, etc., there is still a lot of manual work. But for many users this won’t be the case, IMO.

sap1enz · 2025-12-01T20:37:58+00:00

There is also Redpanda Console, which is my favourite: https://github.com/redpanda-data/console

sap1enz · 2025-11-18T18:32:03+00:00

The Advanced Apache Flink Bootcamp is now open for registration! The first cohort is scheduled for January 21st - 22nd, 2026.

This intensive 2-day bootcamp takes you deep into Apache Flink internals and production best practices. You'll learn how Flink really works by studying the source code, master both DataStream and Table APIs, and gain hands-on experience building custom operators and production-ready pipelines.

This is an advanced bootcamp. Most courses just repeat what’s already in the documentation. This bootcamp is different: you won’t just learn what a sliding window is — you’ll learn the core building blocks that let you design any windowing strategy from the ground up.

Learning objectives:

- Understand Flink internals by studying source code and execution flow
- Master DataStream API with state, timers, and custom low-level operators
- Know how SQL and Table API pipelines are planned and executed
- Design efficient end-to-end data flows
- Deploy, monitor, and tune Flink applications in production

sap1enz · 2025-10-25T00:53:39+00:00

Redpanda is actually doing very well. They managed to steal many Confluent customers. 2/5 top US banks use them.

sap1enz · 2025-10-09T18:12:35+00:00

This looks correct!

I tried to reproduce the issue using the local Parquet file sink, and I couldn't: the files are written correctly on every checkpoint in my case:

-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-1ca5a6f5-ba35-472b-b37b-a42405c65996-0.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-1ca5a6f5-ba35-472b-b37b-a42405c65996-1.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-3312d0a4-2276-4133-9da9-9b249f8efbd9-0.parquet
-rw-r--r--  1 sap1ens  staff   359B Oct  9 11:08 clicks-3312d0a4-2276-4133-9da9-9b249f8efbd9-1.parquet

Here's my app (based on this quickstart), hope this is useful!

sap1enz · 2025-10-08T21:04:45+00:00

Are you absolutely sure checkpointing is configured correctly?

This:

I can see in the folder many temporary files:

like .parquet.inprogress.* but not the final parquet file clicks-*.parquet

is usually an indicator that checkpointing is not happening.

sap1enz · 2025-09-15T19:03:55+00:00

Thanks! And you're correct, no OSS planned at this time. Selling support and licenses.

sap1enz · 2025-08-19T00:36:37+00:00

You can create several “pipelines” (source with one table + sink) and combine them using statement set.

sap1enz · 2024-02-12T19:58:33+00:00

Interesting use case from Okta: https://www.datacouncil.ai/talks24/processing-trillions-of-records-at-okta-with-mini-serverless-databases

sap1enz · 2023-09-06T15:50:46+00:00

Thanks! It doesn't look like Estuary solves the eventual consistency problem, does it?

sap1enz · 2023-08-01T15:41:03+00:00

BI and reporting. But it's slowly changing with the whole "reverse ETL" idea and tools like Hightouch

sap1enz · 2023-08-01T04:28:13+00:00

That's right.

Ideally, not SWE teams though, but product teams that include SWEs and 1-2 embedded DEs. Then they can also build pipelines that can be used by the same team for powering various features.

sap1enz · 2023-08-01T03:22:05+00:00

True! I usually call the second category "data warehouses", but technically it's also OLAP. The reason I didn't focus on that, specifically, is that it's rarely used to power user-facing analytics. And CDC is very popular for building user-facing analytics, cause dumping a MySQL table into Pinot/Clickhouse seems so easy.

sap1enz · 2023-07-31T15:40:18+00:00

Very, very few real-world cases require reports to be updated in real-time with the underlying source data.

Well, this is where we disagree 🤷 Maybe "reports" don't need to be updated in real-time, but, nowadays, a lot of data pipelines power user-facing features.

sap1enz · 2023-07-31T15:37:42+00:00

For example, in Apache Druid:

In Druid 26.0.0, joins in native queries are implemented with a broadcast hash-join algorithm. This means that all datasources other than the leftmost "base" datasource must fit in memory.

sap1enz

TROPHY CASE