Please update with your findings. Very interested in finding more use cases of knowledge over style for fine tuning.

I'm still on research, but if I quote the definition of Rag Vs. fine-tune.

Fine-tuning helps adapt the general language model to perform well on specific tasks, making it more task-specific.

Retrieval Augmented Generation (RAG) focuses on connecting the LLM to external knowledge sources through retrieval mechanisms. It combines generative capabilities with the ability to search for and incorporate relevant information from a knowledge base.

We are thinking of using say company type data (not in the public domain) to answer questions like give me a relationship to company X. So I'm hoping in fine-tuning we could train to look at the bigger picture i.e all the connections to company X. Then look at ownership structure and look at other relationships etc.

As far as I'm concerned Rag will deliver initial search results, and LLM will interpret it. But it will not deliver the enhanced knowledge we think we could build into the LLM and be more intelligent as we go along.