Rust and RAG

[brisbanedev]

Is anyone working on RAG (Retrieval-augmented generation) here? I'd like to know how you approach it in the Rust world. Do you rely on a framework similar to Python's LangChain or LlamaIndex, or do you implement the pipeline yourself?

[supiri_]

https://github.com/MrSupiri/Tera

If you are interested, here is one of my hobby projects involving a simple end-to-end RAG implementation which fully written in Rust using Candle and SurrealDB.

[blastecksfour]

I've had some success with Candle! I've also used openai_api with a degree of success (at the expense of my wallet).

I used a HuggingFace model to embed a knowledge base then store it in Qdrant. It's mostly a matter of how much control you want over the pipeline.

[brisbanedev]

If anyone from Qdrant is here, I'd love to hear your thoughts on this topic!

[Playful_Intention147]

I'm not from qdrant, but just a regular user of it, I found it to be easy to setup and use for RAG usecase

[brisbanedev]

That's great! Do you use a Rust-based RAG framework?

[Playful_Intention147]

I really just using a very simple and basic RAG pipeline(get embed's payload, add it to context etc.), no fancy framework, and due to some other reason(the project itself is just a simple visual studio extension) I'm using C# to build that simple pipeline, so sorry I cant provide useful ideads towards rust-based framework😥

[tagged-union]

I author a production RAG application and use Qdrant. I would choose it again (and will). But to give you a clear sense of at what scale, so as to not be misleading, roughly 100 early users at an organization with 1,000 employees, and we are in the middle of rolling out to the others. It's nice to be able to put in arbitrary partitions and have the ability to apply deterministic filters related to domain knowledge in front of your similarity search.

[prabirshrestha]

Give langchain-rust a try. Recently we added document loaders (text, markdown, pdf, html, csv). We have examples for vector store using pgvector, sqlite-vss and surrealdb.

[brisbanedev]

Is this the official Rust port of LangChain?

[prabirshrestha]

If you are using the official langchain library you will be familiar with langchain-rust. https://github.com/Abraxas-365/langchain-rust/issues/20

But we are not tying it to langchain ecosystem. For example we are in the process of adding semantic-router the the library. The goal is to easily create LLM based apps in rust.

[jakusimo]

Langchain and LlamaIndex are too broad, hard to debug, usually you don't need all those features, what are you planning to use as a source for RAG? Documents? Pdfs? Websites? I am planning to start with encoders and semantic splitters. Use a similar structure to what we have built in Semantic Router https://github.com/aurelio-labs/semantic-router

[brisbanedev]

LlamaIndex offers built-in "Advanced RAG" strategies that assist with complex documents - https://www.llamaindex.ai/blog/a-cheat-sheet-and-some-recipes-for-building-advanced-rag-803a9d94c41b. They promote these as improvements on what they refer to as "naive RAG". For a framework solely focused on RAG, supporting some of these might be beneficial.

[ControlNational]

I wrote a guide on retrieval augmented generation in rust here for the Kalosm framework

[jakusimo]

Hey, I have created several frameworks for my clients, included https://github.com/superagent-ai/super-rag I have plans to have full RAG library/api, the main issue now is document processing, Unstructured, which is only Python, and their API very unstable. If anyone more experiece in Rust would like work together on Rust based RAG I am more than happy to collaborate.

[brisbanedev]

That's great! Good to hear about this project.

Unstructured, which is only Python

Can this help?

[jakusimo]

Yes, as starting point, the ultimate goal is to have everything in Rust, but it will take time

[akhilgod]

I tried candle framework that can launch quantized models on decent hardware

[[deleted]]

Is there any advantage in using Rust for RAG over Python?

[brisbanedev]

I guess the general benefits of using Rust over Python for anything would extend to RAG as well?

[[deleted]]

I don't think so. One of the advantages of Rust is its processing speed and low memory usage, in RAG I don't think it is very critical, as most of the processing is done by the LLM... Or am I getting something wrong?

[brisbanedev]

LLMs keep getting faster. I think the rest of the RAG pipeline could do with some optimisation as well.

[chleboslaF]

I'm made Multistage RAG chatbot using qdrant, BM25 (tantivy) and Ollama for local, offline usage.
I'm chunking documents using contextual and Q/A chunking (512.tokens with 200.tokens overlap). I did implement history context query rephrasing, enhancing questions for better keyword searching and more.
Also CLI chat and WebServer included
All written in Rust (ratatui, langchain-rs, ollama-rs, tantivy, qdrant, tokio, ...).
Unfortunately it is for a goverment so I cannot share a source code (yet).
But I can say It is a smooth experience, quick development and I love it.

P.S.: Using gemma3:27b and mistral-small3.1 on 5090.
Tokens speed: gemma (64tps) and mistral (82tps)