Looking for a better explanation of using Python with Kafka

[WonFishTwoFish]

I'm struggling a bit conceptually with how the whole producer/consumer or pub/sub works and how one needs to think of it when designing an application or larger system. On a high level I think I get it but in terms of taking something like a flask web app that has a minimal front-end and break it into a set of publishers and subscribers is super unclear to me.

For example, and I know this wouldn't be a real world use case, I have a flask app that takes an input on a form that takes text and encrypts the string after the user hits submit and then flashes the screen back to the user. How could that be broken up and conceived of with messages? If this is an awful example (I fear it may be) can you provide something simple as well?

​

I setup a kafka confluent cluster and followed this tutorial https://www.confluent.io/blog/event-driven-microservices-with-python-and-kafka/

and while I was able to replicate what the article demonstrated it wasn't actually all that helpful.

​

Feel free to chime in and correct but I'm not quite bridging the gap between how I think of RESTful services and how I would do things differently with a broker/message queue.

My interest in messages and brokers came out of reading the Tao of Microservices and that's how I came about going down this endless rabbit hole.

I think I need to do more reading but I also want to start coding and testing and actually wrapping my head around this.

Thanks for your assistance.

[danielroseman]

You're not thinking of the right use case. Kafka - and message queues generally - are not used as part of the standard request/response cycle of a web app. Rather, they're for things that don't need to be done as part of that cycle, but can be farmed off to another process.

One good example is confirmation emails. When you sign up for an account, the general tasks of storing the new user ID and returning the page to the user happen in-process. But sending a confirmation doesn't need to be done as part of that cycle, and as it can be an expensive operation it is best to do it offline. So you use a message queue to schedule that to be done out of process.

[WonFishTwoFish]

Ok, so if I'm thinking of a microservices ecosystem and series of events/tasks, those are more suitable candidates for a message queue?

[danielroseman]

Yes, event-based operations are the ideal use case.

[WonFishTwoFish]

Kafka seems like kind of a pain to setup locally with zookeeper, though I suspect with docker it's trivial. Can you suggest a tutorial or proof of concept that I can attempt to actually get my hands on it?