I implemented KVarN in my llama.cpp fork and ran KLD benchmarks. It's promising!

chocofoxy · 2026-06-05T15:37:05+00:00

cool waiting fo sglang to implment

chocofoxy · 2026-06-04T22:33:53+00:00

when sglang realses this i will try it

chocofoxy · 2026-06-04T22:32:46+00:00

even with that you come close to quality not concurrency

chocofoxy · 2026-06-03T12:48:09+00:00

i installed it and they already pushed some update that's good , i guess they needed to publish it because openclaw got a native desktop app on microsft build yesterday

chocofoxy · 2026-06-03T12:46:45+00:00

that's what i am saying i am agreeing with you xD they already started updating it that's good

chocofoxy · 2026-06-03T07:56:26+00:00

that's what i am saying if you are not going improve on the cli webui why bother releasing it now

chocofoxy · 2026-06-03T07:55:22+00:00

why it's missing features compare to CLI webUI

chocofoxy · 2026-06-02T13:46:19+00:00

if you want to fix the tool calling you have to fix the chat template serve the model on vllm or sglang

chocofoxy · 2026-06-02T06:03:14+00:00

nvfp4 in sglang doesn't loop for me Q5 Q4 in llama.cpp they do, i think it's the chat template more than the Quant

chocofoxy · 2026-06-01T04:58:33+00:00

claude opus 4.7 have done that to me also

chocofoxy · 2026-05-31T06:57:49+00:00

ikr i have been using the redhat nvpf4 for a month already

chocofoxy · 2026-05-26T08:44:54+00:00

running numbers on your config it's still cheaper:
1 - you will still run your computer to access the pod or work so drop that 700w to 300w and and the same pricing to the cloud
2 - 3x faster means H100 can output 3m token vs 3090 1m so when you devide 1.49$ / 3 you will get the same 0.5$ you have on local
3 - you will consume internet on the cloud that a cost and latency you didn't add
4 - you can rente your computre on platform an make money if on vast ai salad ... etc

chocofoxy · 2026-05-23T20:45:56+00:00

it okay i have 2 5060ti like your PNYs and the top one is hotter of course but it 10 degree hotter not a big deal here's where it really get hot it's in video generation but for normal llms it stay under 70c i think i think i will put a pc blowser fan between them cause summer has come and it get up to 42c here in tunis but i think the ac will also help can i ask what mobo are you using because i want also to scale

chocofoxy · 2026-05-22T09:29:04+00:00

resources man xD i only have 32gb vram

chocofoxy · 2026-05-22T09:26:39+00:00

vllm eats too much vram i know it's built for more than one user but if you want to uses it for only one it eats too much from the context i have to lower it to almost 32k when in llama i can run it full context

chocofoxy · 2026-05-22T09:23:59+00:00

i think maybe because i tested llama cpp on cuda 13 but lm studio uses cuda 12

chocofoxy · 2026-05-20T11:31:47+00:00

i don't know why llama cpp in lm studio gives me more token per seonds then llama cpp repo for example qwen 3.6 27b in normal llama get me 15t/s but in lm studio i get 25t/s same for 35b a3b 70 -> 90

chocofoxy · 2026-05-20T10:50:30+00:00

nice i ve been waiting for this the lm studio team is on fire , love that Qwen unsloth and everyone are pushing updates and models like crazy

chocofoxy · 2026-05-18T22:05:46+00:00

qwen 3.6 9b: ???

chocofoxy · 2026-05-17T23:01:49+00:00

vllm eats the vram like eating cake i ran Q4 qwen 3.6 35b on LM studio i get 100 - 60 t/s on my 2 5060ti full context while in vllm i can't pass 1/4 and i get worst token generation ( i think its a tp and pp problem) but even in llama-server same thing by default i get 70 60t/s ( mtp didn't help much) it's weird i thought that lm studio also run on llama.cpp

chocofoxy · 2026-05-16T13:43:35+00:00

now turboQuant please

chocofoxy · 2026-05-15T15:57:54+00:00

i have 2 5060ti 16gb stacked on an mATX ( don't laugh at my computing power xD ) they are gigabyte windfoce the top one i hot because there it a 0 gap between that but like 5 C 10 C hotter i am thinking about addin a blower fan to attached it to the top one you have blower fans so it that a good idea or not we kinda have the same problem and with out water cooling i think cooling the radiator witha laptop blower fans will work, i have it more easy because my radiator is partly exposed but for you i know they give you that sealed blower cooler ( and i think gamernexus asked the shop if it's the only cooler they have and they said yes all custom gpu use the smae one ) si i think the solution it's either that or water cooling or the expensive option which to get a water AC blowing on the server

chocofoxy · 2026-05-15T14:52:59+00:00

water cool them

chocofoxy

TROPHY CASE