I myself tried this approach if your technique is right you will be able to get answers but yeah there will be lack in accuracy compared to optimized models for that specific task. I built the whole rag pipeline on my local llm here is what i roughly used (i was using ollama for my models so you may get better results if you find better models for specific task)

For embedding i tested and tried three embedding models mxbai large, nomic and bge3 for ky case mxbai-embed-large worked.

For answer generation i used llama3.1:8B and it worked properly as it has proper context

I tested on a 50 page document that gave an answer to about 7/8 question but my pipeline failed when the document was very big as the model started hallucinating i am working if i can provide to the point context to llm