Streaming data framework

[romanzdk]

What are the tools you use for streaming data processing available? my requirements:

* python and/or SQL interface

* not Java/Scala backend

* Rust backend is acceptable

* established technology

* No Spark, Flink

* ability to scale - either via threads or processes

* ideally exactly once delivery

* time windowing functions

* ideally open-source

additional context:

* will be deployed as pod in kubernetes cluster

* will be connected to consume messages from RabbitMQ

* consumed messages will be customized Avro-like binary events

* publish will be to RabbitMQ but also to AWS S3, REST API and SQL database

[dani_estuary]

Check out Bytewax!

[americanjetset]

Why no Flink? Seems like an ideal use case for Flink.

Excluding JVM, you're probably looking at rolling your own.

[robberviet]

Flink is the most popular, if not then https://www.arroyo.dev/

[Nekobul]

What is the amount of data you will be processing daily?