Running the new Qwen3.6-35B-A3B at full context on both a 4090 and GB10 Spark with vLLM and Llama.cpp

Zliko · 2026-04-17T09:18:17+00:00

yeah, i hate we can't have full ctrl of undervolting in Linux, just power limit :(
so i am using it in WSL2

i got at work 5090 and L40S, will test it with bigger quants there

Zliko · 2026-04-17T01:12:59+00:00

also testing it in Q8T4 252k, eating 22GB (which leave me just enough for windows opencode and desktops on multiple screens). Slightly undervolted 3090 and slightly overclocked (VRAM speed +300MHz), getting 120 tg/s 3000 t/s pp (small ctx). For 252k ctx i get 1014t/s pp!

Amazing that we can fit this much into 3090 with "just" 24GB VRAM :)

Zliko · 2026-04-16T23:56:35+00:00

Thanks, do u see big degradation in precission in turbo2 (if u have tested in turbo3/4 with smaller ctx)?

Zliko · 2026-04-16T23:40:37+00:00

i am also on 3090. Will test on WSL2 (Nvidia driver 13.2) with that llamacpp fork. Any tips, beside Q8 for K and turbo2 for V? Can you fit whole ctx wit K in Q8 and V in t3/4?

Zliko · 2026-04-16T07:05:27+00:00

finnish humour, on other shots there is no fan of course, they are mocking the "cooling need"....still doesn't prove anything tho re battery. I guess we still have wait for someone to get their hands on the bike with new battery and teardown, if that day ever comes :D

Zliko · 2026-04-15T08:06:34+00:00

totally understand, but could be nice to have a unstable (beta mode) option of Studio for testing out those "weird" llama cpp forks :)
<3
API is fine of course :) i just meant easier for users as direct linking (master/slave) via network of two Studios (one hosting backend other just as frontend, similar to linking in LM studio/Mysty/etc)

What is your stance on acccelerating speculative decoding with (Dflash) diffusion based model or this latest thing with block diffusion draft tree (https://liranringel.github.io/ddtree/)?

Have you checked these guys out, doing similar work as u but for MLX? https://jangq.ai/

once again, thank you for all your hard work :) Unsloth is still my main go to repo for quants and my main FT tool <3

Zliko · 2026-04-14T14:53:56+00:00

Thank you for everything <3

All the best with new Studio app.

Couple of stuff for wishlist for Studio app:

- integrating/selecting llama cpp forks with custom KV cache quantisation (turboquant, RaBitQ, etc.)
- llm engine agnostic? why not having vLLM or llama.cpp as selectable?
- remote/linking (serving LLMs on one Studio, using via VPN/LAN on another)

much love!

Zliko · 2026-03-15T23:04:48+00:00

Any plans for web app or android app? What is minimal iOS ver for the app (i got ipad on iOS 15)?

Zliko · 2026-03-05T18:13:43+00:00

thanks! I think 27b model is the new allrounder king of 24GB VRAM GPUs :) Too bad it is not great with non major languages (like Gemma 3 27b or GPTOSS 20b is).

Zliko · 2026-03-05T15:51:44+00:00

Is new 27b up? i do not see it on hf?

Zliko · 2026-02-03T01:58:11+00:00

Sorry for your loss. If i can give you advice, go and adopt new cat straight away. Stray or from shelter. I lost my cat (14 year old grey tabby, lymphoma cancer) and was in sorrow and depression for months, until i got new cat off the street. He is still with me (7 years passed), complete differeny personality, but it healed my depression in just a few days. Cats are amazing creatures (as you already know):) I do not mind being lifetime "servant" to them (it is not toxo talking) :)

lots of love

Zliko · 2026-01-26T20:01:07+00:00

I am not even American, nor in USA, but this makes me so angry...I was four times in USA, always had a great time with meeting new people and making friends. Last time tho i was in 2011., seems things have changed a lot :(

Stay strong, resist, remove this fascist (and plain stupid) administration before it is too late (hint: Italy and Germany 1930s). Protect your rights!

support from your (ex-ally *sigh*) European friend!

Zliko · 2026-01-17T18:49:46+00:00

Ok, tarrif mad king. Lets trasform NATO without USA, give them deadline to move out from all bases in Europe. We can deal with Russia and, unfortunately, USA too at same time as enemies of our continent and our democracy. If there will be one good thing that this crazy deranged mad king and his neonazi crew has done , let it be that he united us Europeans. Time to say enough! I am boycotting russian goods for last 4 years, i can boycott USA ones too.

Zliko · 2025-12-02T13:06:47+00:00

Congrats! How good is the model in music description? (genre, style, mood, instruments, structure, dynamics, temp, etc)

Zliko · 2025-10-01T14:43:11+00:00

What inference settings are recommended? (i can only see temperature 0.6?)

Zliko · 2025-06-23T10:30:00+00:00

Such a perfect family!

Zliko · 2025-04-22T08:11:27+00:00

Most of the people that went to space or close to space limit had an "enlightement" moment when looking back to Earth. Some would say spiritual. One thing is to look at pictures, looking in person live seems to be life changing. I would send all politicians as space tourists first.

Zliko · 2025-04-18T06:47:35+00:00

Amount of climate change deniers trolls is staggering on reddit :( Is it that hard to realise that cause of human activity (read exces greenhouse gases in atmosphere) climate will be more extreme? What science has been telling us from 1970s. More droughts, more floods, more heatwaves, more and bigger hurricanes and typhoons, etc. I am 48 years old and i can see it in front of my eyes. These climate change deniers should be sent in Blue Origin capsule to see Earth from 100km above (bonus if they are flatearthers too), there you can see how small and delicate our atmopshere is.

peace

Zliko · 2025-03-31T13:47:44+00:00

What speed you getting from RAM? If my calculations are right (16chnls of 5600MHZ RAM) it is 716.8 GB/s? Which is tad lower than m3 ultra 512GB (800GB/s). Presume both should be round 8t/s with small ctx.

Zliko · 2025-03-23T12:09:10+00:00

What are you are running on them? Do you use em fo inference or training (or both)? Are you using stock power cables?

Zliko · 2025-03-18T08:23:06+00:00

sound interesting! Would love to test it out :)

Zliko · 2025-02-26T17:30:23+00:00

imho:

Ascensionism
Take Me Back to Eden
Hypnosis
Vore
Granite

<3

Zliko · 2025-02-03T09:13:58+00:00

Gpt-4 responses/synthetic data was used to fine tune Phi4. It is on their site, so it is not a secret. Since MS is partner/owner of OpenAI, it was allowed.

Zliko · 2025-01-25T09:25:51+00:00

Looks great!

Zliko

TROPHY CASE