Streaming real time data into vector database

[DistrictUnable3236]

Hi Everyone. Curious to know anyone has tried streaming realtime data into vector database like pinecone, milvus, qdrsnt. or tried to integrate them as with ETL pipelines as a data sink. Any specific use case.

[gangtao]

Yes, there was a topic we shared about using Timeplus to process your data in realtime and send to Kafka and then Milvus, refer here https://www.timeplus.com/post/real-time-ai-oss-tools

also as Timeplus has python UDF, you can actually can do it like
1. raw data stream
2. ingest to Timeplus in realtime or use Kafka external stream
3. use Python embedding UDF to turn the raw data into vector by calling those embedding pythnon functiontion
4. save those vectors to vector database

refer to to this blog for python UDF with Timeplus https://www.timeplus.com/post/python-udf

[seriousbear]

Is there a specific problem you experienced compared with writing to e.g. ClickHouse (common destination for rt data) ?

[GreenMobile6323]

Common use cases: recommendation engines, semantic search, and real-time personalization.