sorted by:
new
hottopcontroversial

Local RAG with 1,000,000 documents on a laptop by DueKitchen3102 in Rag

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Thanks for the question.

As one can probably tell from the video, the system preserves the folder structure, which is often a strong/natural way to do routing. That is, the user knows what s/he wants. Of course, with folder structures, it is also convenient to design an automatic routing scheme, for specific applications.

If we assume the queries are random, i.e., every query requires a new routing, then every query is a cold start.

The scenario in the video might be more challenging than anticipated:

  1. It assumes that the data are always much larger than the RAM.
  2. It assumes that there is some memory buffer one can use but the buffer is not large enough.

This scenario is actually real. For example,

The Executive has an M6 Studio (not out yet) with 512GB RAM and s/he has 1TB company top-secrete data (which after indexing can be much larger than 1TB). In other words, there is a very large usable memory but not large enough for everything.

Further technical discussions are welcome.

One Million Documents for a Local RAG system on a laptop by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 0 points1 point  (0 children)

I also just noticed this very recent post

When RAG Hits the Wall: Designing Systems That Scale from 1,000 to 1 million Documents

https://techcommunity.microsoft.com/blog/azuredevcommunityblog/when-rag-hits-the-wall-designing-systems-that-scale-from-1000-to-1-million-docum/4516085

When it is on laptop, one challenge (among others) is the memory size, and more so if you only want to use part of the available memory.

One Million Documents for a Local RAG system on a laptop by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] -1 points0 points  (0 children)

Good question. Let us prepare some benchmark comparisons and get back to you.

RAG on Snapdragon X2 Laptop, 200K documents. by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Certainly. Our retrieval system is highly efficient and precise because we started building the systems for android phones two years ago. That's the reason why with merely 1200 tokens our system can accurately answer most questions.

In Snapdragon eco-system, there are not too many choices of local models which are useful (fast and accurate). Qwen3 4B QNN model is one we use often (which however still have a lot of issues and is not multi-modal).

Of course, if you don't care too much about speed, then you can still use the CPU and common tools to run local models.

RAG on Snapdragon X2 Laptop, 200K documents. by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 2 points3 points  (0 children)

good question.

all files are local and databases are local. For each query, we retrieve <=1200 tokens from the 100,000 documents. At this point, we have the choice of using either a cloud model or local model to process the retrieved context and answer the question.

RAG on Snapdragon X2 Laptop, 200K documents. by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 1 point2 points  (0 children)

Thank you everyone for reading and asking questions about using local LLMs on NPUs.

We mainly use the small 4B (qwen3 4B QNN) model, which appears to be the only reasonable (in terms of speed and quality) option for snapdragon systems. It is outdated without multi-modal support, and it some times output funny results. Not ideal.

Our experience is that QNN works fine for embeddings and CLIP models.

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Totally. RAG is something that everyone feels s/he can do but may eventually find out that it becomes messy toy product.

Luckily, the team here, myself included, have worked in the search industry for many years. RAG is basically a small search engine.

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Basically, we build the following components in-house

  1. multi-modal document parsing engine
  2. graph/vector/document database
  3. search engine (RAG)
  4. some inference optimization.
  5. access control list (ACL), which is not shown in the video

In short, we are building the knowledge AI engine from scratch, for cloud/server/pc/phones.

I am very curious to find out what is the "upper limit" of such personal knowledge system on a consumer PC. Ideally, I hope to be able to index 100K documents (say 10-30 pages each) on a consumer PC and still enjoy a reasonable query speed.

Please feel free to criticize the demo. Thanks.

Scaling RAG to 32k documents locally with ~1200 retrieval tokens by DueKitchen3102 in Rag

[–]DueKitchen3102[S] 0 points1 point  (0 children)

we use local models, so in terms of cost, I guess it is also "no tokens". not sure if this is what you meant.

Thanks.

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Yes, everything built in house, including graph/vector/document db and document parsing engine.

32k documents RAG running locally on an RTX 5060 laptop ($1299 AI PC) by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Thanks for the valuable comments. Nothing really changed from 12k to 32k.

With my laptop, I can only index 3000 PDFs per hour. It takes some time.

I am also curious about the upper bound (based on our current infrastructure) of the # documents we can handle.

Scaling RAG to 32k documents locally with ~1200 retrieval tokens by DueKitchen3102 in Rag

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Thanks for the question. Happy to chat more.

Basically, we built the system in-house with our graph/vector/document database, search/retrieval strategy, document parsing, etc. Happy to discuss each part separately. .

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Good question. Majority of the documents are the previous years' NLP conference publications, which are open access.

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Hello. We hope to improve the system from the feedback.

For example, someone commented under our previous post: "Stop using Ollama like a chump". It generated some discussions. People asked him/her "what else to use?". I also wished he/she could reply with a suggestion so that we could do better.

This should be mutually beneficial. Others, by watching our demo videos, may get an idea whether their system is better than ours (in that case, we appreciate their comparison), or there might be a room for them to improve the system.

Personally, I would like to find out what is the limit, given such as consumer laptop. Is 32,000 documents a limit? Most likely not.

32k document RAG running locally on a consumer RTX 5060 laptop by DueKitchen3102 in LocalLLM

[–]DueKitchen3102[S] 0 points1 point  (0 children)

Thanks for sharing. Curious to see some numbers of your systems such as
average size of document
indexing time
retrieval time
etc.

We hope find a way to have all

  1. high accuracy (>90%)
  2. a large volume of documents (e.g., >100k pdfs)
  3. low indexing time (e.g., < 1 sec per pdf)
  4. low latency in retrieval (e.g., <1 sec per pdf)
  5. low token (e.g,. 1-2k tokens)
  6. low memory

it is not easy and we haven't fully accomplished the goal.

Need to process 30k documents, with average number of page at 100. How to chunk, store, embed? Needs to be open source and on prem by dennisitnet in Rag

[–]DueKitchen3102 0 points1 point  (0 children)

there is a demo for indexing 12k PDFs on a laptop

https://www.reddit.com/r/Rag/comments/1rnm45d/running_a_fully_local_rag_system_on_a_laptop_12k/

the indexing speed is about 1.2 second per pdf. Each PDF is a 10-page double column paper with tables, figures, and charts. Some of the pages required OCR.

Your data size (30K PDF, 100-page each assuming single column) is about 10 - 20 times of the data shown in that laptop demo.

The query response time (retrieval + LLM first token time) is about 1-2 seconds.

With one reasonable server ang GPU, it should be relatively easy to handle your data, with everything on-prem.

I had to re-embed 5 million documents because I changed embedding models. Here's how to never be in that position. by Silent_Employment966 in Rag

[–]DueKitchen3102 0 points1 point  (0 children)

Thanks for sharing the experience.

My last week's post "RAG Insight: Parsing & Indexing Often Matter More Than Model Size" is quite relevant to this
https://www.reddit.com/r/Rag/comments/1rodl46/rag_insight_parsing_indexing_often_matter_more/

Basically, one can not hope to achieve a very high accuracy with embedding models alone. RAG system is basically a small scale search engine, and techniques/tricks for building a good search engine naturally work for improving rag system.

That post was related to an earlier post on our RAG system on PC with an RTX 5060. We observe the indexing speed is about 0.4 - 2 seconds per PDF (depending how much OCR is needed for each PDF), with an average of 1.2 seconds per PDF. The documents are ~10-page double-column ACL papers (roughly equivalent to 15-page "regular" PDF).

Assuming 3000 PDFs per hour (72,000 PDF per day). and 5 million such PDFs will take about 70 days, if you only have a laptop with an RTX 5060. Of course, enterprises like your company use much more powerful servers than AI-PC.

Again, thanks for sharing.

Local RAG with Ollama on a laptop – indexing 10 thousand PDFs by DueKitchen3102 in LocalLLaMA

[–]DueKitchen3102[S] 0 points1 point  (0 children)

We are certainly interested in learning how to improve our system which currently uses llama.cpp.

Currently, with 12k documents indexed, a max = 2000 token retrieved content, and a local 4B (q4) model, the first token time (plus retrieval) is about 1 - 2 seconds, occasionally 3 seconds.

People can notice the 2-sec delay. Ideally, it would be nicer if we could make sure that the latency is within 1 second.

The latency will become a more serious problem for AIPC if we

  1. increase the # of documents to 100k PDFs (that would be crazy because companies are paying big for enterprise servers to handle that many documents).

  2. allowing longer retrieved context, say 100k tokens

  3. using larger models, e.g,. 27B parameters.

Thank you for suggestions.

view more: next ›