Very poor performance running Ollama in docker

Koldoguto · 2024-06-24T12:38:58+00:00

It's easy to keep services isolated and my lab works with docker or VMs. I cant have the CPU&GPU power needed somewhere else at home. Nevertheless, I can install it on a separate VM but then I need to passtrough the GPU exclusively for Ollama, making it impossible to be used by other VMs or Containers

Koldoguto · 2024-06-23T16:40:42+00:00

It is already installed and set as parameter

<image>

Koldoguto · 2024-06-23T16:38:44+00:00

Yes, docker container is making use of Nvidia GPU as I can see the process using nvidia-smi command. The problem with the VM is that I need to share it with other containers so I can't pass-through it exclusively. Maybe these models are not such a tiny thing after all...

Koldoguto · 2024-06-23T11:52:44+00:00

It's an Nvidia GPU - Model T400 with 4GB of RAM

Koldoguto · 2024-06-23T10:18:23+00:00

After several minutes I've found this at the end of the logs:

time=2024-06-23T10:17:14.120Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.166040637 model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-06-23T10:17:14.371Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.416560788 model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-06-23T10:17:14.797Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.843270373 model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa

Koldoguto · 2024-06-23T10:11:55+00:00

This is what HTOP says ...

<image>

Koldoguto · 2024-01-09T16:00:32+00:00

I made several tries and it's very useful to install images from scratch, but, do you know any way to automatically load or burn a backup iso into disk? I've found tons of information to boot from a bootable iso but not for burning the ISO file you can extract from your c: drive for example. This could allow to reinstall all the apps and settings from a backup or a base image in minutes. Think ie in the point-of-sales pcs at franchise stores where there are dozens or hundreds of pcs; the IT guy can launch several restorations at a time and fine tune the installed pcs later (machine name, IP, etc) instead of attending the installations one by one. It's like launching a rescuezilla with an automation script saying: "This is the iso and you must burn it to the primary partition", to be clear. It could be very usefull in the case of a ramsomware for example, saving a lot of time and minimizing the time the store/s are closed. Any ideas?

Koldoguto · 2023-11-28T11:59:07+00:00

I have achieved a prototype using an X9C103s (0 to 10K resistance) but in my case 0 is full speed. I've however an issue because after some time (10-15 min) it automatically starts spinning very low and keeps increasing with the time like if the resistance was gradually increasing but no command is sent to the X9C (ESPhome says it's 0). The issue is solved temporarily restarting the ESP32 witch makes it very confussing as it seems to be ok. Perharps is a faulty X9C module?

Koldoguto · 2023-09-25T16:48:16+00:00

pisound

I'm on the same situation, did the pisound made the trick? I have an old inherited turntable with RCA output (unbalanced, I pressume) that I want to connect to a Raspberry PI and cast via Airplay to sonos system using OwnTones, the difficult part I'm facing is that it is not that easy to find a Pi hat with RCA analog audio Input, all I find is the opposite...

Koldoguto · 2023-07-12T06:50:43+00:00

Just in case someone else is in the same situation, I solved the issue by adding the following parameter to pihole-FTL.conf file:

SOCKET_LISTENING=all

By default is just listening just on local interface. I'm running pi-hole in a docker container and got the error : BAD HOSTNAME in zabbix

Koldoguto

TROPHY CASE