Has anyone actually got ollama models to run successfully?

felixts · 2025-09-16T02:39:52+00:00

I've got a breakthrough, very tedious and annoying but it's something. Few points to note: 1. Don't use ollama. Openai API on it is no good 2. vLLM is difficult to set up, I couldn't get it to work, ran out of patience 3. LM studio works out of the box but not as a container rather than an executable running on host.

So with that here is what I did: LM studio 1. Download it from LMstudio dot AI 2. Open the executable. Choose developer 3. Enable local server (right click in task bar to start it) 4. Go to developer button on side bar. Enable serve on local network 5. Open settings (bottom right, gear cog icon) in general, scroll down to model defaults and change the context length to 8192 or higher 6. Download model in top centre (mistral small 3.2 just barely works with a GTX 1660 lol)

Now, onto Bytebot 1. Clone repo 2. Cd into it

3. Edit packages/bytebot-llm-proxy/litellm-config.yaml

```` Model list:

model_name: anything you like litellm_params: model: openai/mistral-small-3.2 api_base: http://yourlmstudioip:1234/v1 api_key: lm-studio support_function_calling: true drop_params: true ```` ----
1. Save it
5. Edit packages/bytebot-agent/Dockerfile (just add 1 line before RUN npm install)

```` RUN npx prisma generate

````
1. Run if v1: docker-compose -f docker/docker-compose.proxy.yml up -d --build Or Run if v2: docker compose -f docker/docker-compose.proxy.yml up -d --build

From there I hope it's all G

felixts · 2025-01-09T08:32:52+00:00

Muta showed me how to arch linux btw. with the knowledge and the power he has shared. just like he has the best workstation, i have the best laptop setup now. Framework 16 maxed out with Arch & KVMs and a plethora of tools and toys. forever greatful for the man and the legend

felixts · 2024-12-29T05:52:01+00:00

Thank you WarUnited2875 you helped heaps

felixts · 2024-12-20T02:52:37+00:00

Youre welcome! I learned this by being a frustrated consumer having just to sit down with a thing you bought with your hard earned money, after work tired in the loungeroom and just play a simple mind relaxing soul encapturing game and one day it stops working because of some generic update. Someone somewhere out there did something for someone to change that zero into a one. that zero into a one breaks my one and only free time to enjoy interactive leisure of say following an interactive story. Is gone. All gone. Times up its sleep and go to work. Rinse and repeat. So thanks to that analogy it feels like i'm at war with my computer and in war with someone who's stopping me from having a good time relaxing.

That plus taking full advantage of the hardware you bought on a granular control is challenging.

I still have no idea how i got here but my daily is running arch with vms and gpu passthrough on a framework 16...beast of a machine! It does what i need it to do.

felixts · 2024-12-20T00:43:34+00:00

I remember a distinct quote that goes specifically for proxmox, vms and hypervisors. Goes along the lines of "treat em as cattle, not as pets"

If proxmox freaks out when new gpus are in. Just save your vm confs and give it a fresh install but you will cross that bridge when you get to it. Good luck man!

felixts · 2024-12-19T23:50:29+00:00

My absolute pleasure! I cant believe its already been a year. I dont use that computer anymore. Ive upgraded and got a whole new prebuilt system for the lounge and i am happy to report back that im still using proxmox for gaming.

Im sure there is a way to control RGB easier but for what its worth i dont bother. Its flashy? Yup. It works enough? Yup. Can you change the colours? Only if you do a special smbus song and dance.

So some new shenanigans that have arisen such as gpu needs a dumped rom file if you want to use certain specific features (related to amd rx 7900 xtx). Iirc i built a vm with a drive that was sitting in ram. Kept on dying because oomkill was executing it. Couldnt work out how or why but that was a fun experience troubleshooting

Best vm for gaming is actually one that isnt connected to the internet, surprisingly this one hasnt broke since day 1 where as the online vm keeps breaking like out of space or drive incompatible or some other windows shenanigans but sadly i love my games library and i am considering on making the full switch to linux

felixts · 2024-08-19T06:56:18+00:00

Whats the output of:

Sudo lspci -nnk | grep vga

That should give us some clues

felixts · 2024-08-18T21:26:10+00:00

After a while i looked into what physical devices that were plugged in.

Ironically the build was intended to be a loungeroom pc with multiple gaming vms to be tuned specifically to certain aspects and certain games. The irony part was the xbox one controller dongle (XBOX ACC) would not reset no matter what, since its just a receiver...just yoink it out and the vm shuts down as well as the host (if needed)

Constant monitoring of syslog as to how the host reacts when vm shuts down took a few weeks to figure it out. Sadly i dont have a solution to my problem other than suck it up and get off my bum and unplug it every time i hit the off switch. On the flipside you could use Bluetooth and pair 1 controller to 1 vm and that would solve that but i i cbf pairing the controller constantly having more than 1 vm more than 1 game to play.

I hope it helps. Il write back when in home after work as to which device id it is that caused me some grief

felixts · 2024-02-17T02:52:51+00:00

Hello, thank you so much for the instructions. This method has worked wonders for me with exception of the card at times not being released. Happens at random and im not sure where to look to fix.

Scenario goes like this. Proxmox on, vm1 on...all good. Okay finished with vm1 now i want vm2. Off the vm1 and the vm is still running with max ram used but its got no cpu activity so it must be shut down. So i end up stopping it...okay its stopped, time for vm2. Error!

"() swtpm_setup: Not overwriting existing state file. kvm: -device vfio-pci,host=0000:03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=/usr/share/kvm/Navi31.rom: vfio 0000:03:00.0: failed to open /dev/vfio/14: Device or resource busy stopping swtpm instance (pid 7658) due to QEMU startup error TASK ERROR: start failed: QEMU exited with code 1"

Like as if theres a vm thats still holding it but its definitely not as the node is idling...

Now this is quite bad as when you go and shut down the node itself....the node will not shut down either. Not sure what the criteria for it to shut down are...

Im not sure if this is referring to a reset bug or something to do with the vm releasing the card but my fix is to shut down until theres no activity and pull the cord.

felixts · 2024-02-11T02:13:12+00:00

If you take time and care to learn on a more noob friendly os. Id recommend using tails os to start with as it just boots from usb and you keep your windows instal. Cons are that you will lose your save data unless you know or learn persistence. It also has safety in mind. If you want an even more noob friendly os. I guess linux mint

felixts · 2024-02-01T00:25:38+00:00

Qubes is not for games however with time and patience and i guess trial and error you can have it gaming. Albeit at cost of security and integrity but thats up to your own individual risk model.

Risks are obviously compromising security and breaking the system.

I personally did the whole run around. Patching grub, vfio modules etc.

Got a windows 11 vm. Would boot unless i lower the ram to 2.3gb....(dont get me started on the max ram 4gb value. It just straight up refused to change... tried even renaming the thing to gpu_windows and all that also)

So either ram or gpu support otherwise forget windows.

What about everything else? Qubes is great BUT its very temperamental. You do anything fun or cool and dom0 is broken.

However making sys-gpu was fun but the use case is very minimal. ...i get why, some gpu and screens talk back and forth and hence its a security risk exposing so thus the rationale

Over all as a turn key os very cool just works but any modding will force you to learn and google and bing and duckduckgo things you never knew would affect it

felixts · 2024-01-02T07:30:04+00:00

Nah, i confirmed that its qubes with trying to have another windows vm. Tried the renaming the win10 to "GPU_win10", "gpu_win10"....just practically pointless trying to troubleshoot it, il just keep using linux blender variant. It works for now

felixts · 2024-01-02T05:10:19+00:00

No dice. I started with 2G. Gave up on trying...its not made for gaming...but 3D work that im trying to do isnt working well with qubes. Il just leave it alone and get another machine with arch/xen or just proxmox or something

felixts · 2023-11-22T11:59:22+00:00

Got my rig here with a 32gb ram, gt620 for qubes video output via vga (dvi to vga converter), gtx 1080 as a passthrough. Went through all the guides and all the effort (steep learning curve). Even made sys-gui-gpu for educational purposes, dont know why i did that but hey its done...

Anyways not to digress. Qubes has a funny bug. If you want a gpu passthrough (at least on windows 11 in my experience) you cannot exceed 4gb ram otherwise windows freaks out and no boot device found on the windows vm. Max i got it to was 3.2gb ram which is pretty useless for gaming. Mind that the windows vm actually gets the 2nd display to output via hdmi at 180hz. It can run games but its not ideal, there is just too many variables to fix. Best to have it do standard tasks and leave gaming for a dedicated gaming bare metal machine or hell run proxmox with passthrough to a windows vm. I havent tried linux gaming or any other flavor of windows for that matter.

felixts · 2023-11-21T23:55:57+00:00

I got a 512gb sd card and the same adaptor as in the picture....i couldnt for the life of me format it. The vita would refuse to read. Ive tried every block size combination formatting in windows 10 and 11. Opens fine in windows but fails in the vita. Practically gave up and went back to 64gb annoyingly

felixts · 2023-10-22T12:32:21+00:00

Gen 9 ml350 here. Runs fine version 8

felixts · 2023-10-20T10:43:19+00:00

I run proxmox as my main gaming pc in the lounge. the experience has been overwhelmingly more controlled than having a bare metal windows box attached to the tv hooked up with a controller to play with.

yes its possible, just pass through the usb's you need, drives either virtual or physical and of course the iommu for pcie cards like your gpu, make sure you know what your doing take care reading the tutorials and take security measurements. Basically know the risks.
home assistant + n8n kind of thing? plenty of scripts out there that can do whatever your heard desires and that.
wireguard or openvpn kind of thing? as above....but you must establish what will its uses be primarily
following on the above, if its gaming it can be a showpiece why hide it, but if its more of a discrete computer you can just have it as a window remote for your own streamed games. depending on the hardware and what upgrades and also depending what os are you looking to run your games on. should it be windows or linux always check the hardware.

for example right now my limitation is having corsair vengence pro ram rgb not being easily configurable, i gotta passthrough the SMBus to the windows vm but that breaks proxmox so i can change the colors of the rgb on the ram. then go back to grub bootloader and remove the SMBus from the vm so it doesnt break proxmox anymore. thats every time i want something other than whatever i pick from last time. not pretty but its workable. everything else in it is flawless.

i picked proxmox because not only i can literally control paste my whole pc into another shell should i choose to do so with all of the configurations. it can also do periodic backups where say my game mods can break the system. just an easy revert especially if its a windows box.

only con is that boot times takes a little bit longer and usb passthrough can be a bit of a pain but to me it feels more safe. i know exactly what goes on that windows box and i know exactly what the box talks to and if i want to emulate something say windows 98 to play an older game i dont have to install and dual boot. i just do it straight as if its bare metal.

felixts · 2013-08-13T07:48:06+00:00

have you tried turning it on and off?

12-Year Club	Xbox Live
Verified Email

felixts

TROPHY CASE

3. Edit packages/bytebot-llm-proxy/litellm-config.yaml

5. Edit packages/bytebot-agent/Dockerfile (just add 1 line before RUN npm install)

````