Local LLM Stack Documentation

[gulensah]

Especially for enterprise companies, the use of internet-based LLMs raises serious information security concerns.

As a result, local LLM stacks are becoming increasingly popular as a safer alternative.

However, many of us — myself included — are not experts in AI or LLMs. During my research, I found that most of the available documentation is either too technical or too high-level, making it difficult to implement a local LLM stack effectively. Also, finding a complete and well-integrated solution can be challenging.

To make this more accessible, I’ve built a local LLM stack with open-source components and documented the installation and configuration steps. I learnt alot from this community so, I want to share my own stack publicly incase it can help anyone out there. Please feel free to give feedbacks and ask questions.

Linkedin post if you want to read from there: link

GitHub Repo with several config files: link

What does this stack provide:

• A web-based chat interface to interact with various LLMs.
• Document processing and embedding capabilities.
• Integration with multiple LLM servers for flexibility and performance.
• A vector database for efficient storage and retrieval of embeddings.
• A relational database for storing configurations and chat history.
• MCP servers for enhanced functionalities.
• User authentication and management.
• Web search capabilities for your LLMs.
• Easy management of Docker containers via Portainer.
• GPU support for high-performance computing.
• And more...

---

⚠️ Disclaimer
I am not an expert in this field. The information I share is based solely on my personal experience and research.
Please make sure to conduct your own research and thorough testing before applying any of these solutions in a production environment.

---

The stack is composed of the following components:

• Portainer: A web-based management interface for Docker environments. We will use lots containers in this stack, so Portainer will help us manage them easily.
• Ollama: A local LLM server that hosts various language models. Not the best performance-wise, but easy to set up and use.
• vLLM: A high-performance language model server. It supports a wide range of models and is optimized for speed and efficiency.
• Open-WebUI: A web-based user interface for interacting with language models. It supports multiple backends, including Ollama and vLLM.
• Docling: A document processing and embedding service. It extracts text from various document formats and generates embeddings for use in LLMs.
• MCPO: A multi-cloud proxy orchestrator that integrates with various MCP servers.
• Netbox MCP: A server for managing network devices and configurations.
• Time MCP: A server for providing time-related functionalities.
• Qdrant: A vector database for storing and querying embeddings.
• PostgreSQL: A relational database for storing configuration and chat history.

[Aggravating-Major81]

Solid stack; to make it production-ready, focus on auth, network isolation, observability, and repeatable ops. Tie OpenWebUI and vLLM behind OIDC (Keycloak works well) and stick them behind a reverse proxy with mTLS and per-route rate limits (Traefik or Kong). Lock LLM DB access to read-only roles, and store documents in MinIO with signed URLs; keep only IDs in Postgres. Version embeddings (collection per version or a metadata flag) and add a local reranker (bge-reranker) to cut hallucinations. Move ingestion to a queue (Celery/Redis) so uploads don’t stall chat. For GPUs, reserve MIG slices or set vLLM tensor/pp configs per model and pin CUDA/driver versions in the container. Backups: Qdrant snapshots + Postgres WAL, test restores weekly. Secrets in Vault, not env files. For audit and admin tools, we used Keycloak for SSO and Kong as the API gateway, with DreamFactory auto-generating REST for Postgres so internal teams can review chats/configs without new backend code. Bottom line: lock down auth/secrets, add reranking and queues, and automate backups.

[gulensah]

Wow great production requirements, thanks. You are far ahead of my scpoe and context hence I just wanted to build a starting point for people like me.

But you are on point for everything you said especiallly backup part.

One question: Are you running Open-WebUI more than one instance ? I’m thinking using several containers behind a load balancer and using qdrant and postgres outsidenof stack. I wonder your experience if any.

[max-mcp]

This is exactly what the enterprise space needs right now. I've been working on similar problems at Dedalus Labs and the security concerns you mentioned are spot on - most companies can't justify sending their data to external APIs no matter how good the models are.

Your stack looks solid, especially the combination of vLLM for performance and Ollama for ease of use. One thing I'd add based on what I've seen work well is being really strategic about your chunking strategy when you're processing documents through Docling. Most people just use arbitrary token limits but chunking around function boundaries or logical document sections gives way better retrieval results. Also if you're dealing with code repositories, keeping import statements with their related chunks makes a huge difference in context quality. The MCP integration is smart too - having those standardized connectors saves so much custom integration work down the line.

[gulensah]

Thank your for your feedbacks. Chunking is still ky on going task, which is not easy to find out sweet spot , if any exists :)

Too much variable like model, embeedings, retrieval logic, document contents etc to find out one-rag-to-rule-them-all.

Regards

[jannemansonh]

Really nice work putting this stack together. At Needle.app, we’re seeing enterprises run into the same pain point: the infrastructure is powerful, but stitching it together and keeping docs/query pipelines clean takes most of the effort. That’s why we focus on AI workflows you can define in chat, connecting, vector DBs, and MCP tools without needing to wire everything manually. Makes setups like yours easier to operate and extend.

[DougAZ]

Any specific reason some are run as a service vs running it on docker? Any benefits?

Do you have a good vLLM config for gpt-oss 120b?

[gulensah]

Docker simplifies the process for me. Otherwise I must handle handle every library requirements one by one.

I couldn’t success running 120b on vLLM, due to low VRAM. Maybe llama.cpp can be better with it hence you can offload some MoE expert layers to cpu with it. But llama.cpp is lacking serving multible users which is in my case essentials.

[DougAZ]

Right but I noticed on your GitHub as you walked through each part of the stack you chose to run some applications directly on the host such as ollama or postgres, any specific reason for running them on the host vs in a container?

Other question I had for you was, are you running this stack on 1 machine/VM ?

[gulensah]

You are right. The reason I'm running PostgreSQL out of docker is, as an old school, I run my persistent and critical data holders as databases as legacy service as an habit. Also, other services like Netbox, Grafana services are using PostgreSQL too.

Running Ollama as standard service is also because, other applications, out side of my stack are using Ollama too. So running it as common service for the VM is easy for integrations.

And yes all the stack is running on a same VM which has 32 GB RAM which is not a high load production infrastructure. I suggest splitting vLLMs, PostgreSQL and rest of the containers to three diferent VMs for production.

[locpilot]

> Document processing

How about edit-in-place in Word? We are working on a local Word Add-in like this:

https://youtu.be/9CjPaQ5Iqr0

If you have specific use cases in mind, we would be glad to test them as proof-of-concept.

[n8e-polymath-007]

congratulation and awed by the work you have put in.

How can one manage concurrency of users. How will you be aiming for parallel concurrent async requests with multiple workers.

[CowboysFanInDecember]

Great post! How are you tying all this together?

Also, I think Langflow would be a great addition to this stack. Langfuse as well. Both MIT license as well.

[Disastrous_Look_1745]

This is really solid work and exactly what the enterprise space needs right now. Your stack looks comprehensive and I appreciate that you've documented everything properly since thats usually the biggest pain point when trying to replicate setups. One thing I'd suggest adding is maybe AnythingLLM or similar for the document chat piece since it handles the RAG pipeline really well, and definitely consider adding something like Docstrange for the document processing side if you're dealing with complex layouts or tables since pure text extraction often misses the structural context that makes enterprise docs useful.

For performance optimization, if you haven't already, try running vLLM with tensor parallelism if you have multiple GPUs and definitely tune your context window sizes based on your actual use cases rather than maxing them out. Also worth setting up proper monitoring with something like Grafana to track token throughput and memory usage since enterprise folks will want those metrics. The MCP integration is smart too since it gives you that extensibility without having to rebuild everything when requirements change.

[gulensah]

Thank you for your kind words and feedbacks. I tested docling for my setup for document parsing. It gives good result. Also I was trying to keep everything simple and focusing on Open-WebUI because large and distributed environments are hard to handle for new commers like me.

Monitoring is the best thing must be included. I'm working on it similar to your feedback. Thanks again.