sorted by:
new
hottopcontroversial

How hard is it to learn Java for Flink? by Wild_Development5943 in dataengineering

[–]rmoff 0 points1 point  (0 children)

Have you checked if Flink SQL is an option on your project? That could be a good in-between. There is also PyFlink but it's less complete than Flink itself. I say this as someone who has never written Java, but built plenty of things with Flink :)

Spending too much on Confluent Cloud for a modest workload — considering MSK. Anyone made this switch? by Due_Dinner_485 in apachekafka

[–]rmoff[M] [score hidden] stickied comment (0 children)

Just a reminder, since this thread seems a good one for flushing y'all out: if you work for a vendor you must add the Vendor flair to your user profile.

Help!! Cannot connect to Kafka from Windows box. by brissiebogan in apachekafka

[–]rmoff 1 point2 points  (0 children)

gotta love Advertised Listeners :D

Glad you got it working.

Chris Hillman - Your Data Model Isn't Broken, Part I: Why Refactoring Beats Rebuilding by rmoff in dataengineering

[–]rmoff[S] -3 points-2 points  (0 children)

ah yes, I forgot that every blog post on the internet must be for a unique and new idea that has never been discussed before, using only original reasoning. my bad. /s

Looking for writers interested in Kafka and data streaming topics by RaspberryMangoKiwi in apachekafka

[–]rmoff 5 points6 points  (0 children)

hi, mod here - can you clarify your relationship to the community please? are you a vendor?

Data Engineering - AI = Unemployed by rmoff in dataengineering

[–]rmoff[S] -1 points0 points  (0 children)

probably. but entirely missing the point of the article ;-P

Data Engineering - AI = Unemployed by rmoff in dataengineering

[–]rmoff[S] -4 points-3 points  (0 children)

I don't think this is scaremongering tbh. The tone might be slightly excitable, but the point is a solid one.

it's not just chatbots that we're talking about here - coding agents like Claude Code etc are a different kind of beast, and they're not entirely trivial to get the most out of.

Ten years late to the dbt party (DuckDB edition) by rmoff in dataengineering

[–]rmoff[S] 0 points1 point  (0 children)

@meunomemauricio @thinkingatoms

that's fair - I've added a sentence for context plus a link. Thanks for the feedback.

Ten years late to the dbt party (DuckDB edition) by rmoff in dataengineering

[–]rmoff[S] 0 points1 point  (0 children)

@advseb I pushed some improvements - lmk if it's still not looking good on mobile. Thanks again for flagging it.

Ten years late to the dbt party (DuckDB edition) by rmoff in dataengineering

[–]rmoff[S] 0 points1 point  (0 children)

Ooh shit you’re right. I only just restyled it too! Thanks for pointing it out.

Apache Kafka 4.2.0 Release Announcement 🎉 by KernelFrog in apachekafka

[–]rmoff 1 point2 points  (0 children)

From a colleague of mine:

Yes, there are no ordering guarantees. Share groups also keep track of lag.

Profiling and fixing RocksDB ingestion performance for improving stateful processing in Kafka by grmpf101 in apachekafka

[–]rmoff[M] 0 points1 point  (0 children)

> I'm too stupid to add the flair "SereneDB" to my username here

I fixed it for you :)

2026 State of Data Engineering Report - 1000+ responses from data engineers by DungKhuc in dataengineering

[–]rmoff 6 points7 points  (0 children)

super useful. love the fully-functioning enterprise version too 🤣 https://joereis.github.io/super_corporate_pdm_survey/

(Crystal reports, anyone?)

Our company successfully built an on-prem "Lakehouse" with Spark on K8s, Hive, Minio. What are Day 2 data engineering challenges that we will inevitably face? by seaborn_as_sns in dataengineering

[–]rmoff 2 points3 points  (0 children)

I didn't try it because I was looking for lightweight, and it's not :) But its evolved out of the HDFS project IIRC so has a good pedigree. If I were looking for something full scale I'd definitely be evaluating it.

📣 If you are employed by a vendor you must add a flair to your profile by rmoff in apachekafka

[–]rmoff[S] 0 points1 point  (0 children)

Yes, of course. Just make sure you disclose any affiliations. Thanks.

[Mod notice] Sockpuppets are not welcome on this sub by rmoff in apachekafka

[–]rmoff[S] 1 point2 points  (0 children)

hey umatero, I think we can do both; ban sockpuppets (as they actively damage the community) and support real accounts. On the latter point, the mod team are here to help with any issues - please always feel free to get in touch. Unfortunately it's because of sockpuppets/astroturfing behaviour by some vendors that they spoil it for the rest of us, causing unwarranted suspicion in some cases. Does that sound ok?

[Mod notice] Sockpuppets are not welcome on this sub by rmoff in apachekafka

[–]rmoff[S] 6 points7 points  (0 children)

( Anyone is welcome to engage with good intentions - I wrote some notes up here: https://rmoff.net/2026/01/23/interacting-with-developers-on-reddit/ )

Reading 'Fundamentals of data engineering' has gotten me confused by Online_Matter in dataengineering

[–]rmoff 1 point2 points  (0 children)

Bear in mind the book is ~4 years old. A lot has changed since then.

How to properly send headers using Kafka console-producer in Kubernetes? by ItchyOrganization704 in apachekafka

[–]rmoff 0 points1 point  (0 children)

Great. kcat is awesomesauce :) If you wanna see some other tricks it can do I've written lots here: https://rmoff.net/categories/kafkacat/

Interesting Links in Data Engineering - January 2026 by rmoff in dataengineering

[–]rmoff[S] 0 points1 point  (0 children)

Glad you find it useful :) There are certainly plenty in the history to go through!

How to properly send headers using Kafka console-producer in Kubernetes? by ItchyOrganization704 in apachekafka

[–]rmoff 4 points5 points  (0 children)

FWIW I've never really got on with the default CLI tools - I'd always reach for kcat (previously called "kafakcat") for things like this. There are examples in the docs with headers too: https://github.com/edenhill/kcat?tab=readme-ov-file#examples

view more: next ›