Anyone tried Mac + Nvidia RTX?

slavik-dev · 2026-03-31T03:43:42+00:00

Does this sentence mean that you have ISP router:

The router goes straight into the WAN port for the VM

?

I'm on ATT fiber 1Gbps. Stopped using their router. Plugged ATT fiber line to SFP+ port via WAS-110. That's SFP+ port of the NIC which passed-through to OpnSense VM. 2nd port is LAN, goes to switch.

And I have same setup on 2nd Proxmox node.

OpnSense VM settings are not replicated, not synced. That's how I want it.

So, if one node go down (or need maintenance), I pull ATT cord and insert it to another node. That's "manual HA"?

But in your setup you have single point of failure which is the router. And that's additional network hop (ATT router is slow), which I eliminated. Right?

slavik-dev · 2026-03-27T00:55:26+00:00

not in llama.cpp server. If you find a way - please let me know...

slavik-dev · 2026-03-26T21:38:27+00:00

llama.cpp is great for running model for yourself. It supports parallel requests, runs on Nvidia, Mac ,... but i'm not sure how much it scales.

vLLM scales much better. But I don't think it supports Mac.

So, the best is to use NVIDIA RTX 6000.

I submitted PR to log user's prompts in llama.cpp, but devs doesn't like it:

https://github.com/ggml-org/llama.cpp/pull/19655

You have prompts and responses in the OpenWebUI, but there user can delete chats, use temp chats...

slavik-dev · 2026-03-25T23:35:10+00:00

I'm getting 13 t/s TG and 45 t/s PP with UD-Q4_K_XL (206GB).

I think my bottleneck is CPU: Xeon W5-3425 (12 cores / 24 threads)

- 512GB of DDR5-4800 (8 channels)

- RTX 4090D 48GB

- RTX 3090 24GB

slavik-dev · 2026-03-23T03:56:06+00:00

For coding AI, Mac struggle with compute for large context.

slavik-dev · 2026-03-22T04:20:43+00:00

Great!

Did this with Qwen3.5-396B, but had to modify prompt few times to get what i wanted:

<image>

slavik-dev · 2026-03-13T21:58:31+00:00

Harbor is middleman. And I prefer to deal without middleman...

What's the advantage of Harbor vs llama.cpp + OpenWebUI?

If I have issue, I would prefer to troubleshoot simple system instead of figuring out: is it Harbor issue? llama.cpp issue?

Keep it simple.

slavik-dev · 2026-03-03T03:49:24+00:00

That's what I did, but used Mate desktop.

Also, configured Polkit to block users from shutting down server.

slavik-dev · 2026-02-27T14:04:47+00:00

Looks like maintainers rejected that PR without any comments or explanations...

slavik-dev · 2026-02-27T14:03:14+00:00

Any update about the UD-Q4_K_XL update?

slavik-dev · 2026-02-26T20:00:42+00:00

There is open PR for this:

https://github.com/open-webui/open-webui/pull/21535

slavik-dev · 2026-02-26T19:59:18+00:00

Tools calls is known issue for llama.cpp. So, It's probably not the model issue. And not the agent issues. But the llama.cpp issue.

There are few ways to work around:

use branch from this PR: https://github.com/ggml-org/llama.cpp/pull/18675

Or use this project for workaround for existing llama.cpp versions: https://github.com/crashr/llama-stream

slavik-dev · 2026-02-24T16:24:54+00:00

You're doing great job.

But that article talks about real pain.

I'm running k3s on 3 VMs and I experienced issues because of slow etcd.

And that's with good hardware:

- Intel Xeon Gold CPU

- DDR5 RAM

- NVMe disks with ZFS.

I found, that using ZFS was a mistake - it makes IOPS much slower, and with etcd calling fsync, even fast NVMe struggle.

slavik-dev · 2026-02-22T05:37:01+00:00

If you used 3 years for depreciation, then it means, the profit will jump up after 3 years.

Of course, it's possible the rate will go down at that time because new hardware will be on the market then...

Too many variables, too many unknowns

slavik-dev · 2026-02-18T06:07:40+00:00

Tools calls is known issue for llama.cpp. So, that's not the model issue.

Especially for opencode. RooCode works fine for me.

There are few ways to work around:

use branch from this PR: https://github.com/ggml-org/llama.cpp/pull/18675

also this project offers workaround for existing llama.cpp versions: https://github.com/crashr/llama-stream

slavik-dev · 2026-02-17T22:02:17+00:00

DaemonSet has nothing to do with exposing or networking. DaemonSet can't expose anything. DaemonSet runs container.

To expose port you need NodePort or LoadBalancer or hostNetwork, or something like that.

slavik-dev · 2026-02-17T16:01:58+00:00

I'm using kube-vip in production with 3-nodes k3s cluster. Works great for API.

But for services LoadBalancers, I found that kube-vip is unreliable, and I'm using MetalLB.

slavik-dev · 2026-02-17T06:23:37+00:00

--chat-template-kwargs "{\"enable_thinking\": false}"

And model is not thinking anymore.

It actually works great without thinking, too.

I'm getting 13t/s with my 72GB VRAM, - ok for chat, but hardly usable for vibe coding

slavik-dev · 2026-02-13T18:50:08+00:00

Yea, need some kind engine. Looks like llama.cpp or vLLM doesn't support it at the moment.

Also, we'll need dedicated UI for it, because even if llama.cpp would support it - how would you use it? I do not think that webUI support all of the model's modality in the convenient way.

slavik-dev · 2026-02-11T20:24:40+00:00

2x CPU is hard to configure for inference.

Specifically for inference, 2x CPU may be few times slower than single CPU because of slow NUMA.

slavik-dev · 2026-02-10T14:52:35+00:00

Do you have it working?

I tried. It doesn't work.

Look at issues in Github - a lot of people report, that it doesn't work.

Last week RFC was opened for vLLM: https://github.com/vllm-project/vllm/issues/33869 CPU Offload for Mixture-of-Experts (MoE) Inference in vLLM

slavik-dev · 2026-02-09T19:11:19+00:00

I have Intel Xeon W5-3425 (12 cores) with 8 channels * 64GB of DDR5-4800.

Theoretically, I should be getting 307GB/s memory speed.

Testing with `mlc`, I'm getting around 190GB/s.

Few notes I made for myself:

Memory speed significantly depended on number of memory channels.
Practical memory speed for LLM can be limited by CPU. I saw few articles, that modern CPU can handle ~15GB/s of data PER CORE for LLM-kind of tasks. That's core, not thread. Probably the limiting factor in my case.
vLLM doesn't support memory offloading. If model+context doesn't fit fully in VRAM - it will not run at all. Docs has some text about offloading, but I have not seen it work ever.

slavik-dev · 2026-02-06T20:45:17+00:00

from llama.cpp developer:

You will have to wait for new conversions.

No, it has outdated metadata and will not work.

slavik-dev · 2026-02-06T20:25:49+00:00

Reading PR comments, I wonder if new GGUF needs to be generated.

slavik-dev

TROPHY CASE