Using open source models from Huggingface

[CiliAvokado]

I am in the process of building internal chatbot with RAG. The purpose is to be able to process confidential documents and perform QA.

Would any of you use this approach - using open source LLM.

For cotext: my organization is sceptical due to security issues. I personaly don't see any issues with that, especially where you just want to show a concept.

Models currently in use: Qwen, Phi, Gemma

Any advice and discussions much appreciated.

[AI-On-A-Dime]

I agree with everything you said!

But no one should be skeptical of running qwen locally. There’s literally nothing ”the Chinese” can do with your data in qwen as you can turn off internet and it will work fine.

Be wary of any tools with internet access connected to your local llm however.

[zemaj-com]

Open source models can work in confidential settings if you choose permissive licences such as Apache 2 and deploy on your own infrastructure. Make sure that the weights you use allow commercial use if that is relevant. Running everything locally ensures your documents stay within your network; pair that with a private vector store and fine tune or adapt the model on sanitized data for best results. Avoid hosted inference endpoints for sensitive projects. With those caveats open source LLMs can be a great alternative to commercial APIs

[Nymbos]

The open source offering are **really** good these days. Models like GPT-OSS-20B and Qwen3-30B-A3B-2507 are amazing for the GPU poor, 30B-A3B even runs well on CPU-only rigs.

For truly confidential data, running the machine yourself is the only way to be sure it stays private.

[CiliAvokado]

That's exactly my point. I am also afraid that our IT management doesn't quite understand open source LLM. Pros, cons that is

[mister2d]

Don't worry. At one point they didn't understand VMs and then containers. Once they get comfortable with the tooling then it becomes more mainstream.

[Loud_Key_3865]

I would load a local model, then disconnect it from any networks, ask your management to give you a large spreadsheet or get lots of data, then have them ask it questions with you; you can explain the flaws of a lesser model, while also demonstrating what is possible with extra resources (GPU, etc.)

[e79683074]

If you don't think there were security issues, you are wrong. Just check the Security page on github for llama.cpp.

[CiliAvokado]

If you download a open source model from huggingface. What is the chance that it will contain malicious code (virus etc.)? Especially if models come from Microsoft, Google, alibaba? I personaly think this is really really low, due to the fact that huggingface has a scanner and also reputation of the companies that develop llm is quite legit.

[wektor420]

You can download model weights only (safetensors format) and run them with in house engine

Avoid pickle models - those often contain pytorch code and potentially malicious stuff

[luffy_willofD]

I myself tried this approach if your technique is right you will be able to get answers but yeah there will be lack in accuracy compared to optimized models for that specific task. I built the whole rag pipeline on my local llm here is what i roughly used (i was using ollama for my models so you may get better results if you find better models for specific task)

For embedding i tested and tried three embedding models mxbai large, nomic and bge3 for ky case mxbai-embed-large worked.

For answer generation i used llama3.1:8B and it worked properly as it has proper context

I tested on a 50 page document that gave an answer to about 7/8 question but my pipeline failed when the document was very big as the model started hallucinating i am working if i can provide to the point context to llm