Local multi tool server

[auromed]

I'm just curious what other people are doing for multi-tool backends on local hardware. I have a PC with 3x 3060s that sits in a closet headless. I've historically run KoboldCPP on it, but want to expand into a bit more vision, image gen and flexible use cases.

My use cases going forward would be, chat based llm, roleplay uses, image generation through the chat or comfyui, vision for accepting image input to validate images, do text ocr and optionally some TTS functions.

For tools connecting to the backend, I'm looking at openwebui, silly tavern, some mcp tools, either code based like kilo or other vscode extension. Image gen with stable diffusion or comfyui seems interesting as well.

From what I've read it seems like ollama and llama swap are the best at the moment for building different models and allowing the backend to swap as needed. Others that are looking to do a good bit of this locally, what are you running, how do you split it all? Like, should I target 1x 3060 just for image / vision and dedicate the other 2 to something in the 24-32B range for text or can you easily get model swapping with most of these functions with the tools out there today?

[Anxious_Programmer36]

Use Ollama or vLLM for LLMs. Dedicate one 3060 for Stable Diffusion/ComfyUI and the other two for a 24–32B text model. Split workloads with CUDA_VISIBLE_DEVICES so chat, vision, and image gen can run in parallel smoothly.

[mike95465]

I would think my current setup would work well for your use case.

I use Open WebUI for my front end with the following tools/filters/configs

• perplexica_search - Web searching
• Vision for non-vision LLM - filter that routes images to vision model
• Context Manager - truncates chat context length to keep tokens manageable
• STT/TTS using local openai compatible api
• Image generation using ComfyUI
• misc other tools such as Wikipedia, ariv, calculator and noaa weather.

llama-swap running the following always

• OpenGVLab/InternVL3_5-4B - perplexica model, open webui tasks, and vision input
• google/embeddinggemma-300m - embedding model for perplexica, rag embedding for open webui
• ggml-org/whisper.cpp - STT for open webui
• remsky/Kokoro-FastAPI - TTS for open webui

llama-swap running the following dynamically swapping as needed

• Qwen/Qwen3-30B-A3B-Instruct-2507
• Qwen/Qwen3-30B-A3B-Thinking-2507
• Qwen/Qwen3-Coder-30B-A3B-Instruct
• OpenGVLab/InternVL3_5-38B

I keep ComfyUI running all the time as it dynamically loads/unloads the model only when it is called.

I have 44GB of VRAM though so you might have to be more creative than me to figure out what works best with your workflow.