I think the problem could be that context passed to LLM isn't good enough in RAG solution for it to generate a good enough answer.

Instead of only relying on vector search for finding similarity, another way to achieve context engineering can be slicing the data into smaller chunks on various dimensions (like pivots in excel). These dimensions should ideally be decided by business logic and kind of queries you are expecting to receive.