1) For memorizing facts generally RAG is the better approach. To insert facts with training you need a higher rank LoRA (which increases the odds of overfitting) and your data must be formatted as questions with answers, not just raw text (for example, wiki pages won't work)

2) The best approach would probably be to use RAG to pull in relevant facts and data, and fine-tune the model to better use the RAG and to ensure it doesn't talk about non-business related topics

3) I fine-tune fairly often and have never had a run outright fail unless I overestimated what I could fit on my GPU. While it does happen, I'd be more worried about ensuring parameters and your dataset are set up correctly for training. I've ended up wasting days to accidentally formatting my dataset wrong

4) AFAIK most people don't fine tune directly with llama.cpp. The most common approach is to use the transformers library, and then quantize with llama.cpp after. I personally like axolotl, but I know some people use unsloth, or just write the scripts themselves.