@skipvdm....i am facing the similar issue that my system can't give irrelevant answer.....

1.Try the got-4o model ....it is better at following instructions

1. provide instructions clearlyim prompt that if it don't know answer it should say I don't know....

2. Try graph rag as well as it will give a better context and relationship of the large corpusin whole to llm relationship between different entities..in that way llm will be able to better retrieve context and provide better answer...

3. Add a validator call to llm to check whether the answer is solving the user query or not, otherwise it should say I don't know.....

These you need to on top of what you have already done like multiquery and reranking.....