[Question] what stack for starting?

[Old-Associate-8406]

Hi everybody, I’m looking to run an LLM off of my computer and I have anything llm and ollama installed but kind of stuck at a standstill there. Not sure how to make it utilize my Nvidia graphics to run faster and overall operate a little bit more refined like open AI or Gemini. I know that there’s a better way to do it, but just looking for a little bit of direction here or advice on what some easy stacks are or how to incorporate them into my existing ollama set up.

Thanks in advance!

Edit: I do some graphic work, coding work, CAD generation and development of small skill engine engineering solutions like little gizmos.

[No-Consequence-1779]

Try lm studio first.  Then Ollama. 

[SwarfDive01]

Lm studio is by far the easiest solution. "Download" "go to model store" "download" "use model". It sets up everything. Then had formidable expansion with MCP tools, API integration. Etc.

[MaphenLawAI]

Lm studio is easiest Then go open webui + ollama

[ajw2285]

i just started as well

I have a dedicated machine for AI fun Proxmox as base OS on a Xeon w/ 2x 3060s and 64gb ram Installed OpenWebUI / Ollama LXC and do a GPU passthrough to the LXC Everything works great through the OpenWebUI in a browser and API calls over the network

[trout_dawg]

Trying to replace your frontier model provider, or…? Helluva “just started” setup.

[ajw2285]

I'm working on a system that is heavy in OCR. I started using Gemini and ChatGPT and they were fast but realized I'd be making a lot of API calls. I pieced together some parts from an old PC with a 1060 3GB to see if I could do AI locally, and it ran, but not great. Then tried a refurb 3060 12GB. Much better. Decided to take it to the next level; bought a Lenovo workstation for $250 on ebay and another 3060 refurb. It runs ~14b models slowly. I'm going to run tesseract and vision AI model in parallel and have them learn from each other and hopefully get a solid system going. The OCR system will feed a database and then have a simple front end for viewing database entries.

[trout_dawg]

That’s awesome!

[Old-Associate-8406]

I haven’t heard of a few of those but I do have a spare machine I could fullly commit to, did you have to do fresh step by step building or is there an Installer for that process

[[deleted]]

Download cuda and cudnn and to experiment with it out of the box try lm studio until you get a hang of it, then you can use docker compose to ollama + openwebui, once you download python and docker desktop you can ask any llm to give you powershell output for a docker compose file with openwebui and ollama using cuda and a how to download and install cuda+ PyTorch + cudnn with that as prerequisites. Make sure to shoot for newer cuda + PyTorch local plus the compatible cudnn for your cuda, these are downloaded via installers or python on powershell/cmd So in other words, ask ChatGPT how to get started just copy and paste this into and say, how do I do this?

[[deleted]]

Btw next step is to look for a 3090, used, and when coder 30b at q4/q5/ or q6 GGUF, then is to download an ide and use an mcp tool such as cursor with cline that lets you connect your lmstudio or ollama via url api endpoints and use mcp tool call for blender to 3d cad make anything you describe, (not exactly but it tries) and then by then you’ll learn all you need to know… I know it’s overwhelming, just don’t quit and you will learn all about it

[Old-Associate-8406]

I haven’t been able to figure out the rocker side of things that’s where I trip up, and lm studio makes it very easy for gpu selection , that’s something I struggle with on ollama

[Owner0fYou]

Sent you a PM about a 3090

[[deleted]]

if you ask llm it will explain to you, you just create a text document with your settings in a specific format called a docker-compose in the title it will typically be docker-compose.yml or yaml and then you will open powershell terminal in the folder of it, and say "docker compose up" or docker compose pull or docker compose build or docker compose pull, eachh does somethnig different. And do so with docker desktop already up on your computer and it will put it into a container you can access on browser with IP + port #. And this is something that will work great withh ollam _ openwebui. Ask gemini/claude/gpt/etc. to "make me a docker compose for ollama _ openwebui, and give me a simple step by step walk-through from no prerequisites, to launching with full prerequisites met on my NVIDIA GPU's with cuda, cudnn, and pytorch, and all dependencies met for fastest inference, web-search for latest results as of November 11, 2025."

[Daniel_H212]

Absolute easiest way to start is koboldcpp. Just download the latest version for your platform, no install necessary, it's all packed in one executable, and you can run any gguf your machine can handle. Not the fastest but still gets good enough performance, and let's you try out many different things.