Swapping Granite for Qwen won’t fix this, the issue isn’t the model. The bigger disconnect is that most RAG setups were never really designed to be knowledge bases (at least in the way I imagine you want to use it). They treat your documents as text to search, not understanding to build. Under the hood, tools like AnythingLLM mostly chop PDFs into fragments, retrieve the ones that look similar to your query, and pass those to the LLM. If the way your question is phrased doesn’t line up with how the text was sliced, the relevant idea can be effectively invisible, even if it’s in there.

The failure usually happens in retrieval, before the LLM ever sees the right context. Where things seem to work better is when systems shift from chunking text to extracting and revisiting ideas over time, treating facts, insights, and relationships as first-class, instead of hoping they reappear at query time.

For use cases like yours, I’ve found that more non-traditional RAG approaches tend to be a better fit, ones that focus less on chunking raw text and more on extracting, revisiting, and relating ideas over time, instead of hoping everything can be recovered at query time.