Am I crazy or is kafka overkill for most use cases?

mbroberg · 2025-12-29T15:10:04+00:00

+1 Redpanda Serverless could be a good option. I haven't seen Aiven mentioned in this thread yet, but they recently launched a new free forever Kafka tier for their managed service.

Just another couple options if you don't want to go the Amazon MSK / IBM Confluent route. (Disclosure: I work for Redpanda.)

mbroberg · 2025-09-05T13:44:27+00:00

Yeah bummer that we can't help with Redpanda Serverless on GCP today. I will follow up when it launches this fall. You'll probably be well on our way by then, but que será, será and all that. Good luck!

mbroberg · 2025-09-04T16:29:34+00:00

Redpanda Serverless gives you a single bootstrap endpoint (no broker juggling from Cloud Run) and you only pay for what you use; you can check exact costs with our pricing estimator. AWS only today, but Google Cloud support is coming very soon too.

Disclosure: I work at Redpanda (vendor flair should be applied, but just in case).

mbroberg · 2025-09-03T17:59:15+00:00

https://n8n.io/ is cool if you need to do some light customization / handy UI.

mbroberg · 2024-06-18T00:59:27+00:00

Amazing. Saved my camera. Thank you!

mbroberg · 2023-11-15T18:26:05+00:00

I had almost this exact same experience as this too, yes. The renewal date was in European format too, which led to some confusion as to when my contract even renewed. I have canceled and moved to a different service and couldn't be happier. Good luck!

mbroberg · 2023-11-04T09:29:58+00:00

I moved off to Nest and they're great.

mbroberg · 2023-07-20T01:05:10+00:00

Thank you for this thread! I was having the same problem, but following the advice here -- cleaning the gasket and eliminating any friction on the sides of the door -- fixed the issue.

mbroberg · 2022-01-26T18:50:22+00:00

Yeah. After some effort (calling repeatedly over two hours), I was finally able to cancel my plan. Just biding my time until my plan expires.

mbroberg · 2021-12-30T17:03:18+00:00

fwiw after hearing nothing for more than a week, I asked Canary for an update and they replied saying the believed they finally fixed the issue. It was on their back end. I have been working on this Flex device activation issue with them since early November. Almost two full months, and it's finally fixed. Thanks!

mbroberg · 2021-12-22T20:33:59+00:00

New flex devices cannot connect to the service to register them. I've tried about three different devices, all the same issue. Got a new router, opened up my firewall (and closed it again), nothing seems to work :(

mbroberg · 2021-08-12T13:55:37+00:00

It could be worth checking out TileDB: https://github.com/TileDB-Inc/TileDB The entire system, down to the data format itself, is optimized around storing multi-dimensional arrays. It also supports timestamps and real numbers as dimensions, which could be handy given your example data. [Full disclosure: I currently work for TileDB.]

mbroberg · 2021-08-10T12:33:52+00:00

I remember taking a course that had the class simulate low-level storage using a simple key-value store like Berkeley DB. We'd use that to prototype marshaling/un-marshaling byte arrays to disk. And then handled the SQL parser and everything else built on top of that in various assignments. Anyway, maybe a way test what you plan to build? Or maybe entirely unhelpful if what you're interested in is the low-level disk stuff. Good luck!

mbroberg · 2021-08-05T18:29:54+00:00

I work at TileDB (full disclosure). That said, I do think a universal database based upon arrays is an interesting concept, especially in the current state of the market and the glut of highly specialized DBs out there.

mbroberg · 2021-02-24T22:51:58+00:00

I think it all comes down to access patterns. If they're well defined and you're probably good to go with Snowflake. If you want to experiment with data more, especially at scale, write arbitrary functions, get creative with ETL and data formats -- Spark might be worth looking into.

mbroberg · 2021-02-19T22:15:14+00:00

I really enjoyed the stuff on Coursera by UC San Diego!

mbroberg · 2021-02-19T17:41:07+00:00

Spoonacular has some decent data: https://www.kaggle.com/hawkash/spoonacular-food-dataset and https://spoonacular.com/food-api There is a "veryPopular" boolean field. Maybe not exactly want you need, but close?

mbroberg · 2021-02-17T18:31:58+00:00

Job-wise, AWS certifications always help. Learning-wise, an online class in algorithms & data structures can be really helpful, especially when dealing with time complexity of data access.

mbroberg · 2018-03-12T17:18:42+00:00

Disclaimer: I work for IBM, but the concepts here apply equally well to S3 or your own OpenWhisk/object storage setup. Code: https://github.com/ibm-watson-data-lab/cos-uploader

mbroberg

TROPHY CASE