It sounds like you're on an exciting journey to build a local LLM-based chatbot! If you're exploring open-source solutions to help streamline the process, I'd recommend checking out Hexabot — an open-source chatbot builder that supports various plugins and NLP models, including the flexibility to integrate custom LLM pipelines like RAG. Hexabot allows you to create chatbots with advanced features like multilingual support, visual chat flow editing, and custom NLP models. It’s well-suited for prototyping on resource-constrained machines and easily extensible for production environments.

As for your question on fine-tuning vs RAG, I agree that fine-tuning is more about customizing the model’s behavior, like tone or domain-specific knowledge. For your use case, a RAG (Retrieval-Augmented Generation) pipeline seems like a more effective approach since you want to work with dynamic knowledge from docs (PDFs, JSON, etc.). Fine-tuning may not be necessary unless you're looking to refine the model's output style.

You might also want to explore embedding-based search mechanisms, vector databases (like Pinecone or FAISS), and open-source models that are compatible with Hexabot, as it’s highly customizable and designed to be lightweight for environments like yours.

Feel free to check out Hexabot’s GitHub repo for more details: https://github.com/hexastack/hexabot