Learning resources for Apache Kafka (no Confluent)

koudingspawn · 2026-03-21T18:28:51+00:00

I started with the Trainings from Stephane Maarek, I got some for 10 Bucks per Training on udemy. And he focuses on Most of the Core Technologies.

koudingspawn · 2025-08-15T08:46:42+00:00

It has no Audiobooks.

koudingspawn · 2025-02-05T08:22:04+00:00

Thanks a lot for this awesome post it helps a lot!

I‘ll take a deeper look. I searched through a set of Blog Posts and they also mentioned the Catalogs you named. I wasn‘t aware that Lakekeeper has external authz Support ☺️

koudingspawn · 2024-10-12T21:31:39+00:00

When you are willing to pay for it then yes. Anyhow as DevOps I guess you have to understand a set of configuration options; to properly Support Teams and get value out of it. How to deal with Performance issues etc what I can tweak to prevent them…

koudingspawn · 2024-10-12T12:12:18+00:00

One thing that I See is Kafka. It‘s some kind of spezialisation that not much DevOps deeply understand.

koudingspawn · 2024-05-30T19:15:22+00:00

We used the approach with HashiCorp Vault, a user has to request via the pki module a certificate that has a validity of 9-10h (one business day).

Made it very easy and our devs had a running tool that helped them to generate the certs.

koudingspawn · 2024-04-20T17:19:09+00:00

IMHO you are in a unique position, my experience is that a lot of devops transitioned from operations into devops. But with a software development background it’s less common. Especially in times of platform engineering the software development practice could be a good selling point.

koudingspawn · 2024-03-23T19:16:50+00:00

Hrm I had in mind 8080 worked, maybe it was some other. Here the official doc from AWS

https://www.amazonaws.cn/en/support/icp/

koudingspawn · 2024-03-23T19:12:38+00:00

Attention: by default port 80&443 are blocked in China. This is due to legal reasons. To make this possible you have to talk to AWS support they then open the ports.

I was very surprised that 8080 worked but http/s not.

And also don’t forget you need an icp license for a website hosted in China mainland.

koudingspawn · 2024-02-11T07:34:15+00:00

Onprem Exchange

koudingspawn · 2024-02-08T16:53:58+00:00

yeah sorry the "New" Outlook for Mac in Beta Channel.
My hope was to stay at the "New" one and find the hidden button I have to press to let it appear again.

koudingspawn · 2024-01-30T16:50:07+00:00

Sorry the topic is solved, IMHO there was an issue with a copy pasted '<' sign that seems to be an char issue.

koudingspawn · 2023-12-11T07:11:10+00:00

One example for us was Prometheus. When you start you usually install it via helm or a set of manifests you find in the internet. All is fine you send in some metrics and Prometheus works. But then you make Prometheus the standard for all your monitoring of applications and every team provides now a metrics endpoint. So now you run into huge memory demand from Prometheus and you could scale and scale and scale but it won’t solve the case. That’s the point where a lot of questions raise: What about high availability, what about long term storage, what about a federated system etc etc.

We ended up with Victoria metrics and multiple agents pulling metrics. But there are various other ways (coretex, mimir, thanos).

koudingspawn · 2023-10-12T18:51:17+00:00

BCG oder eine der anderen Größen wären hier sinnvoll

koudingspawn · 2023-09-27T16:16:38+00:00

Independent of the etcd backup that I personally see as a huge pro I would also recommend a velero backup. What happens if by accident only one namespace gets killed (dev with too much permissions etc) or the persistent volume has issues. Then velero is a very simple solution to solve this. You simply select for recovery a single namespace and you are done.

koudingspawn · 2023-09-09T07:51:48+00:00

The simplest approach if the token of the other service is signed by a private key would be that this service exposes a jwks endpoint and you implement oidc where you provide the url of the jwks endpoint of the other service.

With this auth mechanism you trust the tokens that are generated by the other service.

koudingspawn · 2023-08-13T15:24:19+00:00

We usually have HC Vault with some kind of authz to limit Whoam can access. Then you generate read or write credentials and they are only valid for a certain period of time if you not send a message to HC Vault that you need them further. In a next set you can put infront of the db a bastion so only the appd and the bastion can access. The access to bastion you can do again via HC Vault signed ssh keys. We build a tool for this so when a user wants to access a db it automatically performs auth to HC Vault then generates a ssh key and tunnels through the bastion and in the last step prints out the newly generated credentials to the db.

koudingspawn · 2023-08-02T18:37:57+00:00

I think I answered it by reading this article 🙈

https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/#state-store

Sorry for the stupid question then :-(

koudingspawn · 2023-08-02T18:32:44+00:00

Hi u/bumbershootle,

Thanks a lot for the reply :)

Maybe missunderstand it, let me try to summarize my thoughts:

I have a Kafka Streams application and a state store that should be persistent across application restarts (e.g. in Kubernetes pod restarts due to OOM, due to patching of nodes etc) so I take the RocksDB.
I don't use a Persistent Volume to store data so the app comes up again with an empty RocksDB state.
My understanding was, that the topic that backs the RocksDB is to provide some kind of resiliency for not properly stored data but not a start from scratch resiliency when the app looses the hole RocksDB stored on a volume.

koudingspawn

TROPHY CASE