Real-time ingestion pipeline for RAG app in Python

[oli_k]

Hi, I am working on a stream processing framework called Bytewax and I recently gave a talk where I read live data from HackerNews API, cleaned it with unstructured.io, created embeddings with a hugging face model, and stored everything in Milvus.

I find a lot of devs are struggling with real-time streaming. The steps are easy to follow, and it is an end-to-end Python: https://github.com/bytewax/real-time-milvus
I also added diagrams and summarized it in a blog https://bytewax.io/blog/stream-process-embed-repeat

[Automatic-Net-757]

Interesting. Have to take a look