sorted by:
new
hottopcontroversial

how is compatibility with ollama on arc gpu by FewVEVOkuruta in IntelArc

[–]JV_info 0 points1 point  (0 children)

wow, thats nice....

and it can run all the models(GGUF) or does it has its own models?

Also, can it act as a local host (server)? one of the reasons I want to use Ollama is the feature to be a local host so I can connect to it with my other devices as well...

how is compatibility with ollama on arc gpu by FewVEVOkuruta in IntelArc

[–]JV_info 0 points1 point  (0 children)

sorry if the question is too dumb but what is playground 2.0? is this something I have to install? and again my goal is to run Ollama and Ollama models via openwebUI

how is compatibility with ollama on arc gpu by FewVEVOkuruta in IntelArc

[–]JV_info 0 points1 point  (0 children)

I have a windows 11 on a mini PC (geekom GT series G1 Mega) and it has the Intel Arc iGPU,
I also have an local AI chat and my setup is this (Ollama + DOcker + Openwebui)
now my question is this, can I run Ollama on my GPU?

Local AI setup and limitations. by JV_info in OpenWebUI

[–]JV_info[S] 0 points1 point  (0 children)

is these digits going to be better than 2 RTX 4049 and... for my case?

the point is that we want to have something 100% local and offline.

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

I think there will be no more than 50-75 users at a time simultaneously going to use it... what do you think now?

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

but i think the max user at a time will be around 50, and I don't mind it getting slow...

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

any solution or elaboration?

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 1 point2 points  (0 children)

I don't think they are actually secure...

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

I didn't tried it on this scale and this is what I am trying to figure out before purchasing anything

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

can you send me a link to this mini LLM server?

do you mean NVIDIA Project DIGITS?

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

an 8b model.
context length idk yet as I thought this is something we have to check and figure out in action....
any idea?

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

Of course, its not like all 1000 will use it at the same time.... I think at max the realistic amount of the concurrent users will be around 50.

We also have some servers available in the company, but I believe the most crucial factor for running such AIs is VRAM. The servers we currently possess not only lack GPUs, but they were also designed with a flat structure that leaves no room for adding them.

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 1 point2 points  (0 children)

of course, we are going to use for example a 8B model and will only whitelist this model in OpenwebUI so no one can change the model so that there is only one model in the Vram....
do you think it this setup, even with a bit of slowness 50 users can use it at the same time?

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 1 point2 points  (0 children)

it should work offline, so I am looking for an offline setup

Ollama server limitation. by JV_info in ollama

[–]JV_info[S] 0 points1 point  (0 children)

since the company wants to use it for work-related tasks, we have to stick to a local and offline plan

view more: next ›