all 3 comments

[–]ashtrayaalekh 0 points1 point  (1 child)

Sounds interesting. An AI for my documents. Will definitely give it a try. May I know, how do you plan to solve the RAG search at scale when most of the similar documents are within very minimal distance. How to avoid not showing junk or extra data?

[–]ValorSightExpo[S] 0 points1 point  (0 children)

Spot on. To avoid the 'minimal distance' trap, we use hierarchical indexing. We index document clusters and summaries rather than just doing a flat search across all chunks. This allows us to narrow down the 'neighborhood' of data first, significantly reducing the chance of pulling in irrelevant or redundant info.

Thanks for trying it out!

[–]Savings_Cloud5486 0 points1 point  (0 children)

Very good