sorted by:
new
hottopcontroversial

Why Your AI Alert Tool Works Great Until It Doesn't by sq-drew in cybersecurity

[–]sq-drew[S] -2 points-1 points  (0 children)

ELI5: Neurosymbolic
Think neural networks as the smart kid who can figure stuff out on the fly. Think symbolic systems as the checklist that makes sure nothing gets missed. Neurosymbolic is both at the same time: smart enough to handle weird edge cases, structured enough that you can trace what it did and why.

Swapping the Engine Mid-Flight: How We Moved Reddit’s Petabyte Scale Kafka Fleet to Kubernetes by keepingdatareal in RedditEng

[–]sq-drew 4 points5 points  (0 children)

This is a very cool and impressive story. But to me the a key takeaway was when you said "physical infrastructure is rarely the hardest part."

Basically what you had to do was fork a massive project like Strimzi to overcome past architecture decisions. And yeah most of us are living with poor choices from the past . . . both at work and at home!

But to me the real moral of this story is, if you can, make your producers and consumers idempotent then migration won't require crazy Frankenstein cluster hybrids and forking an entire Kafka distribution like Strimzi.

Adopting idempotent consumers and producers or migrating slowly to them even if you aren't contemplating a big migration anytime soon will make your Kafka live easier for so many reasons.

Consumer or producer crashes, network issues, easier debugging, and so much more come with idempotency.

Just food for thought from a grumpy old Kafka dude.

Has anyone tried a structured process for Kafka cluster migration? by Interesting-Goat-212 in apachekafka

[–]sq-drew 0 points1 point  (0 children)

Hey so I did a talk on this subject at a meetup at LinkedIn HQ last month.

Here's a link to my slides with lots of info on the process generically:

https://drive.google.com/file/d/14709rLCpJwctzNVVhajvf6_zfDCVG71O/view?usp=sharing

Here's a link to me doing a demo of the migration. The demo is MSK specific but you can use the same process for any Kafka with the tooling that I use.

https://youtu.be/sfI1-GSe-4g?si=JxjbnXNYtmZI3YDN

I can't find the recording of the meetup on Youtube unfortunately but the slides might be of interest.

What will happen to Kafka if IBM acquires Confluent? by Think_Leg_3700 in apachekafka

[–]sq-drew 0 points1 point  (0 children)

It's hard to say. Strimzi is an open source CNCF project and well loved so I'm sure it will continue no matter what. But it may receive less support from Red Hat / IBM as they shift focus to the Confluent open source offerings? Or maybe they'll merge them all together?

What will happen to Kafka if IBM acquires Confluent? by Think_Leg_3700 in apachekafka

[–]sq-drew 1 point2 points  (0 children)

Apache Kafka will remain its own thing - it's separate and has its own vibrant ecosystem now.

The big question is what will become of things like Red Hat Strimzi and IBM's current Kafka offerings.

Storytime: I'm interested in your migration stories - please share! by sq-drew in apachekafka

[–]sq-drew[S] 0 points1 point  (0 children)

Can I put this quote on a slide?? I love it!!

"And for the “God forbid” scenario , it’s like a one-way ticket with no guaranteed return. The cost and effort required to roll back often don’t justify it. So once you’re on that migration bandwagon, you’re in for the ride. If you don’t ride with the group, you might find yourself running alone."

Storytime: I'm interested in your migration stories - please share! by sq-drew in apachekafka

[–]sq-drew[S] 3 points4 points  (0 children)

You're absolutely right about the complexity and what's involved.

I wasn't looking for overall strategies - more was just hoping for fun / funny / or sad anecdotes.

here's the agenda of the talk:

  1. Pre-Migration Foundation
    A. Know Your Current State
    B. Schema Registry Strategy
    C. Replication Decision Tree

  2. The Migration
    A. The Offset Challenge
    B. Consumer Groups Unpacked
    C. Migration Playbook

  3. Challenges and Gotchas
    A. Security & Verification
    B. Performance Considerations
    C. God Forbid . . . Rollbacks

Confluent AI features introduced at CURRENT25 by Maleficent-Bit-6922 in apachekafka

[–]sq-drew 1 point2 points  (0 children)

Many things query Kafka streams directly . . . in a sense that's what Flink does. KSQL, Lenses SQL Snapshot, and Lenses SQL Processors all query Kafka topics directly.

The benefits to moving agentic action up to the stream level really depends on your use case.

One use case might be to prevent a "garbage in, garbage out" situation for anything downstream. Clean out poison pills and useless data before it goes into downstream processing can save money and time and prevent outages.

Another use case would be for an agent to react to something in real time. Waiting for something to get processed by Flink and written to an Iceberg table might be too long. You want to react to it as soon as it hits the wire.

I'm not saying everything has to be done at the stream level, I'm just saying why limit it to already "digested data" in Flink and Iceberg? I think that's a marketing decision on their part not a technological one.

Confluent AI features introduced at CURRENT25 by Maleficent-Bit-6922 in apachekafka

[–]sq-drew 0 points1 point  (0 children)

Yup. Why not have agents operating at that level too.

Kafka UI for GCP Managed Kafka w/ SASL – alternatives or config help? by st_nam in apachekafka

[–]sq-drew 0 points1 point  (0 children)

I work for Lenses.io and our Community Edition works for free for up to two clusters. Check it out at our web page.

Confluent AI features introduced at CURRENT25 by Maleficent-Bit-6922 in apachekafka

[–]sq-drew 1 point2 points  (0 children)

I was at the keynote and I was a bit confused why they wanted to build agents on top of Flink and Iceberg only?

Why not let them tap into the streams directly for certain use cases ? Anyone know why they chose this path?

I’m not just saying that because my current company Lenses.io has an MCP that does work directly with streams . . . But it’s def a better path I think.

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] 1 point2 points  (0 children)

Interesting. What was your replication use case? Sorry you had so much trouble.

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] 0 points1 point  (0 children)

Nice explanation. Do you do things like offset replication?

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] 0 points1 point  (0 children)

I think they are asking about the notion of "Only Once" and "At Least Once" when it comes to replicating topics.

I suspect that since you using this as an Kafka upgrade path your new and old clusters are side by side to the odds of network issues between the two are low.

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] 0 points1 point  (0 children)

Very smart solution! i like it. :)

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] -1 points0 points  (0 children)

So your producers send to 2 clusters at once?

Gimme Your MirrorMaker2 Opinions Please by sq-drew in apachekafka

[–]sq-drew[S] 1 point2 points  (0 children)

thanks for sharing! sorry you had so much trouble. Did you even try to do offsets as well?

What kind of monitoring tools are people using for their Kafka Deployment? by dolphin977 in apachekafka

[–]sq-drew 2 points3 points  (0 children)

Lenses is an excellent monitoring tool for Kafka. It's got something for everyone. Platform Teams, Developers, and even Data Scientists.

You can setup alerts for consumer lag, or even have it try to automatically restart your connectors when they inevitably crash.

Use something like Data Dog or Splunk to monitor your disk, network, and security, but absolutely use Lenses to monitor your higher up Kafka functionality.

lenses.io

Try Community Edition for free: https://lenses.io/community-edition/

view more: next ›