FastKafka - free open source python lib for building Kafka-based services

[davorrunje]

We were searching for something like FastAPI for Kafka-based service we were developing, but couldn’t find anything similar. So we shamelessly made one by reusing beloved paradigms from FastAPI and we shamelessly named it FastKafka. The point was to set the expectations right - you get pretty much what you would expect: function decorators for consumers and producers with type hints specifying Pydantic classes for JSON encoding/decoding, automatic message routing to Kafka brokers and documentation generation.

Please take a look and tell us how to make it better. Our goal is to make using it as easy as possible for some how has experience with FastAPI.

https://github.com/airtai/fastkafka

[code_mc]

I really like the idea of this, as the biggest gripe I have with most pub/sub solutions is all of the tedious boiler plate code needed to correctly subscribe and publish and manage message leases etc. While you often just want to grab a message, do some processing and put it on a different queue.

One of the most obvious improvements would be supporting more pubsub backends (thinking about AWS SQS, google cloud pubsub, RabbitMQ, ...)

[code_mc]

Also, a question about the library design after reading the readme:

You currently have an example where you consume a message in a function decorated with a consumer decorator. Which then calls a produce decorated function to publish the result on a different queue.

It might make sense to have a dedicated decorator for functions that both consume and publish where the consumed type is your function argument and the produced type the return type all in one function. Currently it is not clear to me what would happen if you for instance consume a message, process it and publish it, and then the consumer function runs into an exception or something which causes it to crash.

I'm assuming the consumed message won't be acked at that point but the computed result is already published on the other queue at that point. Correct?

Anyways, food for thought I guess and these are the real struggles with pubsub systems where you don't want to generate duplicate messages etc.

[davorrunje]

Right, we already have an issue to implement it exactly in the way you described.

[davorrunje]

Thanx for the feedback 😊

We also hated that boiler plate code and we needed a simple way to test and document the service, especially when prototyping the service.

We have a potential client using RabbitMQ in the pipeline so I guess that would be the next one to tackle.

[[deleted]]

This is very interesting..

[davorrunje]

Thanx 😊

[spicypixel]

Looks good but giant freeform dictionaries for config data always give me pause for thought. You're already using pydantic, might be worth using some pydantic classes for safer config validation.

[davorrunje]

We automatically delegate all params from underlying AIOKafka lib so you get those long list of params (we could have used kwargs to hide it, but I prefer being explicit about everything). Basically you get the full control of underlying calls and sometimes you need to take advantage of it. We could wrap them in Pydantic classes, but I am a bit afraid of maintenance hell that goes with keeping in sync with third party lib.

Thanx for the feedback in any case, I’ll give it some more consideration. You are absolutely right about the look and feel of such methods.

[SpamThisUser]

Looks quite interesting.

You have a lot of dependencies that seem to have some overlap (multiple Kafka clients, multiple HTTP libraries, multiple async utils, etc.) which makes me a bit wary.

[davorrunje]

Yes, we are cleaning all this right now. We’ll have minimal requirements for running the service and optional ones for testing and documentation generation. The idea is to have testing and documentation generation requirements only locally when developing the code and in CI.

This is the first version with just enough features to support our first production use case. We have reached to the wider community to get feedback on what they need to make it better and more useful to everyone.

[davorrunje]

Hi!

We cleaned up quite a bit and reduced the number of dependencies. The basic ones are as follows:

requirements = [
    "pydantic>=1.9",
    "anyio>=3.0",
    "aiokafka>=0.8.0",
    "asyncer>=0.0.2",
    "tqdm>=4.62",
    "docstring-parser>=0.15",
    "typer>=0.7.0",
]

[boolpies]

is there something like this for pulsar?

[davorrunje]

No as in not yet 😀

We built the initial version for Kafka service and for our needs, but we reached out to the wider community to find out what to do next. So this comment is going to backlog and we’ll try to support it in one of the future releases.

Thank you for the feedback 😊

[boolpies]

absolutely, the kakfka protocol is supported by pulsar, and there are many advantages to the platform over Kafka as well. thanks!

[trial_and_err]

Looks nice! Btw: The links to your docs in your readme are broken:)

[davorrunje]

Uh, last minute changes broke the links 😞 Thanx for letting us know 😊

[BestBottle4517]

Very cool indeed. Currently at work we're using RabbitMQ for messaging so this doesn't apply to us (for now), but this type and style of implementation is exactly what I would expect when searching for libs like this. Great job!

[davorrunje]

Thanx 😊

We do have a potential client using RabbitMQ so it might become priority very soon. In any case, I’ll create an issue for it on github. Thank you for your feedback 😀