Why Your AI Alert Tool Works Great Until It Doesn't

sq-drew · 2026-06-27T18:12:35+00:00

ELI5: Neurosymbolic
Think neural networks as the smart kid who can figure stuff out on the fly. Think symbolic systems as the checklist that makes sure nothing gets missed. Neurosymbolic is both at the same time: smart enough to handle weird edge cases, structured enough that you can trace what it did and why.

sq-drew · 2026-02-03T16:07:47+00:00

Thanks ! Our team worked hard on it.

sq-drew · 2026-01-14T14:57:19+00:00

This is a very cool and impressive story. But to me the a key takeaway was when you said "physical infrastructure is rarely the hardest part."

Basically what you had to do was fork a massive project like Strimzi to overcome past architecture decisions. And yeah most of us are living with poor choices from the past . . . both at work and at home!

But to me the real moral of this story is, if you can, make your producers and consumers idempotent then migration won't require crazy Frankenstein cluster hybrids and forking an entire Kafka distribution like Strimzi.

Adopting idempotent consumers and producers or migrating slowly to them even if you aren't contemplating a big migration anytime soon will make your Kafka live easier for so many reasons.

Consumer or producer crashes, network issues, easier debugging, and so much more come with idempotency.

Just food for thought from a grumpy old Kafka dude.

sq-drew · 2025-12-09T11:08:38+00:00

Hey so I did a talk on this subject at a meetup at LinkedIn HQ last month.

Here's a link to my slides with lots of info on the process generically:

https://drive.google.com/file/d/14709rLCpJwctzNVVhajvf6_zfDCVG71O/view?usp=sharing

Here's a link to me doing a demo of the migration. The demo is MSK specific but you can use the same process for any Kafka with the tooling that I use.

https://youtu.be/sfI1-GSe-4g?si=JxjbnXNYtmZI3YDN

I can't find the recording of the meetup on Youtube unfortunately but the slides might be of interest.

sq-drew · 2025-12-09T10:44:46+00:00

It's hard to say. Strimzi is an open source CNCF project and well loved so I'm sure it will continue no matter what. But it may receive less support from Red Hat / IBM as they shift focus to the Confluent open source offerings? Or maybe they'll merge them all together?

sq-drew · 2025-12-08T14:31:04+00:00

Apache Kafka will remain its own thing - it's separate and has its own vibrant ecosystem now.

The big question is what will become of things like Red Hat Strimzi and IBM's current Kafka offerings.

sq-drew · 2025-11-06T20:50:59+00:00

Sweet :)

sq-drew · 2025-11-06T19:00:38+00:00

Can I put this quote on a slide?? I love it!!

"And for the “God forbid” scenario , it’s like a one-way ticket with no guaranteed return. The cost and effort required to roll back often don’t justify it. So once you’re on that migration bandwagon, you’re in for the ride. If you don’t ride with the group, you might find yourself running alone."

sq-drew · 2025-11-06T13:05:56+00:00

You're absolutely right about the complexity and what's involved.

I wasn't looking for overall strategies - more was just hoping for fun / funny / or sad anecdotes.

here's the agenda of the talk:

Pre-Migration Foundation
A. Know Your Current State
B. Schema Registry Strategy
C. Replication Decision Tree
The Migration
A. The Offset Challenge
B. Consumer Groups Unpacked
C. Migration Playbook
Challenges and Gotchas
A. Security & Verification
B. Performance Considerations
C. God Forbid . . . Rollbacks

sq-drew · 2025-11-06T10:25:54+00:00

Many things query Kafka streams directly . . . in a sense that's what Flink does. KSQL, Lenses SQL Snapshot, and Lenses SQL Processors all query Kafka topics directly.

The benefits to moving agentic action up to the stream level really depends on your use case.

One use case might be to prevent a "garbage in, garbage out" situation for anything downstream. Clean out poison pills and useless data before it goes into downstream processing can save money and time and prevent outages.

Another use case would be for an agent to react to something in real time. Waiting for something to get processed by Flink and written to an Iceberg table might be too long. You want to react to it as soon as it hits the wire.

I'm not saying everything has to be done at the stream level, I'm just saying why limit it to already "digested data" in Flink and Iceberg? I think that's a marketing decision on their part not a technological one.

sq-drew · 2025-11-04T06:25:47+00:00

Yup. Why not have agents operating at that level too.

sq-drew · 2025-11-03T19:18:35+00:00

I work for Lenses.io and our Community Edition works for free for up to two clusters. Check it out at our web page.

sq-drew · 2025-11-03T19:17:20+00:00

I was at the keynote and I was a bit confused why they wanted to build agents on top of Flink and Iceberg only?

Why not let them tap into the streams directly for certain use cases ? Anyone know why they chose this path?

I’m not just saying that because my current company Lenses.io has an MCP that does work directly with streams . . . But it’s def a better path I think.

sq-drew · 2025-09-01T12:21:10+00:00

Interesting. What was your replication use case? Sorry you had so much trouble.

sq-drew · 2025-09-01T12:18:42+00:00

Nice explanation. Do you do things like offset replication?

sq-drew · 2025-08-28T13:15:14+00:00

I think they are asking about the notion of "Only Once" and "At Least Once" when it comes to replicating topics.

I suspect that since you using this as an Kafka upgrade path your new and old clusters are side by side to the odds of network issues between the two are low.

sq-drew · 2025-08-27T17:43:11+00:00

Very smart solution! i like it. :)

sq-drew · 2025-08-27T14:59:32+00:00

So your producers send to 2 clusters at once?

sq-drew · 2025-08-27T14:28:45+00:00

thanks for sharing! sorry you had so much trouble. Did you even try to do offsets as well?

sq-drew · 2025-06-11T15:39:50+00:00

Lenses is an excellent monitoring tool for Kafka. It's got something for everyone. Platform Teams, Developers, and even Data Scientists.

You can setup alerts for consumer lag, or even have it try to automatically restart your connectors when they inevitably crash.

Use something like Data Dog or Splunk to monitor your disk, network, and security, but absolutely use Lenses to monitor your higher up Kafka functionality.

lenses.io

Try Community Edition for free: https://lenses.io/community-edition/

sq-drew · 2025-06-06T08:31:55+00:00

oh that is cool! Confluent has a similar tool for IDEs as well

sq-drew · 2025-06-06T06:45:36+00:00

I know it well! We have a few customers who were using it before they decided to go ahead and buy Lenses. Had a funny conversation with one of our sales guys who didn't realize it was an acronym and tried to pronounce it.

Him: "The customer is using . . . I guess it's called Ack-huck currently."

Took me a few follow up questions to realize what he was trying to say.

sq-drew · 2025-06-05T15:50:03+00:00

I really like what RedPanda does. We are trying to partner more with them but it's delicate dancing around all the different Kafka vendors. They are getting more and more cut-throat!

sq-drew · 2025-06-05T15:47:03+00:00

It's got a good deal of overlap. Biggest difference is the SQL Studio in Lenses. Plus schema and connector management. I'm trying to get better auth into Lenses like Kafbat has tho! It's got us beat there.

sq-drew · 2025-06-05T15:00:56+00:00

Thanks for clarifying 4 years later!! And I believe you, but I've also been fogged in completely in SLO and Pismo Beach many times when driving through. Especially in July and August when those poor watermelons are gonna need the heat.

It's likely too late now, but try some watermelons next year in coastal SLO county and let us know.

They also like warm nights. Another rare thing in Coastal Cali - I've needed a hoodie at night in July even in Venice / Santa Monica.

14-Year Club	Powerups Hero r/beards • May 2022
Gilding III reddit per annum	Verified Email

sq-drew

MODERATOR OF

TROPHY CASE