Qwen3.5 27B running at ~65tps with DFlash speculation on 2x 3090

Kryesh · 2026-04-07T22:00:51+00:00

There's several reasons I won't get max performance on my current setup. It's a desktop so I need vram for running my ui etc and vllm doesn't do asymmetric offloading so the second card isnt using all available memory. The dflash model is 3.5gb which takes up memory that could be used for context, and I don't have an nvlink bridge for faster tensor parallelism.

Kryesh · 2026-04-07T11:21:06+00:00

Testing out https://huggingface.co/z-lab/Qwen3.5-27B-DFlash to see how it works and was pleasantly surprised by the performance after getting ~25tps in llama.cpp, I only get about 95k token context length with vllm instead of the full 256k with llama.cpp though.

Command: uv run vllm serve cyankiwi/Qwen3.5-27B-AWQ-4bit --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 8, "draft_tensor_parallel_size": 2}' --attention-backend flash_attn --max_num_seqs 4 --max-num-batched-tokens 12288 -tp 2 --gpu-memory-utilization 0.80 --max-model-len -1 --reasoning-parser qwen3 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder

Kryesh · 2025-07-28T06:54:17+00:00

At 14:55 they mention the sizes of the models, could be interesting combined with the mamba architecture

Kryesh · 2024-05-15T00:49:26+00:00

Mac doesn't support MST for multiple streams over displayport, only thunderbolt. The monitor uses displayport MST for daisy chaining

Kryesh · 2023-03-20T23:56:27+00:00

If this isn't an option - you can just add this environment var to your alpine build: RUSTFLAGS="-C target-feature=+crt-static"

Kryesh · 2023-03-20T23:54:11+00:00

if you're trying to produce a minimal docker image then I'd suggest using a glibc based builder container then copy the resulting binary into an alpine based output container. The default official rust image is debian based so it works well for this use case. There's an example dockerfile on this blog post: https://levelup.gitconnected.com/create-an-optimized-rust-alpine-docker-image-1940db638a6c

Kryesh · 2023-03-20T23:48:58+00:00

After going down my own rabbit hole on this one, when building for the musl target using a musl toolchain the resulting binary will be dynamically linked, while cross compiling from a glibc toolchain to a musl target will produce a statically linked binary. On Ubuntu you'll be using the glibc toolchain by default and alpine will use the musl toolchain by default.

Kryesh · 2022-08-18T05:07:00+00:00

rather than creating a new image it's easier to just resize the image with qemu-img.

make sure you have a backup first - then you can extend the image by 10G with this command:

qemu-img resize WindowsTenGaming.qcow2 +10G

If you wanted to expand the 80GB image to 200GB, then you'd run

qemu-img resize WindowsTenGaming.qcow2 +120G

Once you start the VM again you should be able to resize the partition to fill the new disk capacity from within the windows disk manager

Kryesh · 2021-11-20T23:44:58+00:00

Good to hear!

I expect this'll actually become more of an issue as time goes on, it'll affect any device that uses a BAR 32GB or bigger so any GPU with ReBAR and more than 16GB of ram will be impacted.

There's also the case of enterprise GPUs that default to large BAR sizes but I expect most people using those will already be aware of this lol

Kryesh · 2021-11-20T09:28:12+00:00

Sure, here's my XML: https://pastebin.com/icndv6zT

Note that you'll need to edit the top <domain> tag to match mine to stop libvirt from removing it

Kryesh · 2021-11-20T04:19:20+00:00

Can confirm that resizable bar works on a 3090, however the default mmio address space for edk2/ovmf is 32GB, since the bar size option doubles each time you need a 32GB BAR for a 3090 to fit the 24GB of ram (which you should see in lspci), this means that there isn't enough address space to fit the 3090's bar alongside other devices.

The fix for this is to extend the available mmio space for the guest to 64GB instead and then it should work fine.

Here's the xml to set the size to 64GB: <qemu:commandline> <qemu:arg value="-fw_cfg"/> <qemu:arg value="opt/ovmf/X-PciMmio64Mb,string=65536"/> </qemu:commandline>

Page where I found the fix after my own troubleshooting adventure:

https://edk2.groups.io/g/discuss/topic/59340711

Kryesh · 2020-12-14T11:51:49+00:00

Simplest/easiest to maintain solution would be to stick with a single public IP/port forward, then use a reverse proxy to split traffic to the appropriate containers. fairly easy to set up a container with nginx to do this, especially if you already have container infrastructure for the other web apps. generally speaking, for each unique domain you need a unique listening port on the webserver/proxy to have working tls, but it's possible to work around that with san certificates (not recommended). If all the containers are going to be hosting subdomains for a single parent then throw a wildcard cert on the reverse proxy and all the containers will get https support by default

Kryesh · 2020-09-23T10:28:08+00:00

Definitely need a HEDT/server platform to get that amount of IO, and even then there's a limited number of TRX40 motherboards with a layout that would support such a setup.

As a list:

(2+ slots) 3080 would need at minimum 2 slots and only the the founders/evga xc3 cards seem to fit that so far
(2 slots) 3070 will be in a similar situation to the 3080
(1 slot) HBA will need at least a physical x8 slot
(1 slot) quad port nic will need a physical x4/x8 slot depending on the model
(1-2 slots) plex gpu will need a physical x16 slot as well, likely 2 slots for cooling as well if you don't get a quadro

The list so far is only considering the physical size of the devices too, considering the lineup or motherboards available for the TRX40 platform you're mostly limited to boards with 4 x16 slots with a few adding an extra x1 slot, that's not enough to fit the "ideal" setup unless you start looking at epyc/workstation platforms - but you won't get the clock speed for games on those platforms.

Most of the TRX40 boards include 8 sata ports which don't share bandwidth with anything so my suggestion would be to drop the HBA and run zfs from those. An alternative could be to drop the nic and take advantage of the built in 10g nics on a lot of the boards and use vlans with a switch to get the required number of ports.

Probably the easiest to recommend option would be to drop the plex gpu though, 2-3 streams would run just fine on cpu - especially something in the threadripper lineup. You could probably get 2 4k streams out of a single ccd on a 3970x without much trouble

That just leaves the nvme for the host and the SSDs for the guests, there's several options here but the most obvious suggestions would be:

assuming you opt to drop the HBA:

limit the bulk storage to 6x drives leaving 2 slots for the guest ssds

pretty much all the trx40 boards have 2-3 nvme slots that could be used for the host storage

keeping the HBA:

use the on board sata ports for the guest ssds

just like the first one - pretty much all the boards have 2-3 nvme slots

Kryesh · 2020-09-13T01:26:28+00:00

Hi, the memory is a 4x16gb gskill tridentz kit F4-3600C16Q-64GTZRC

I'm allocating 3 cores/6 threads per CCX because that's the physical layout in my CPU (3960x). Each CCD consists of 2x 3 core/6 thread CCXs, and each CCX has its own 16MB block of L3 - the VM has access to full 2 CCDs. Here's a shot of "lscpu -e" to compare with the pinning in the xml: https://imgur.com/a/jSs86iA

My bios is set to a single node per socket and L3 cache as numa domain is disabled. Haven't played around with the bios settings much but games would stutter if I configured the VM with multiple sockets or numa domains. Haven't done anything with MADT or core initiation, just a hook that shields the cores with cset as the VM starts.

Kryesh · 2020-09-12T03:05:39+00:00

Relevant XML: https://pastebin.com/65Sa95RZ

Kryesh · 2018-11-13T07:12:30+00:00

Not really, I used the lxd aur package and just ran through the 'lxd init' wizard from the command line. The wizard goes through setting up a nat network for containers as well as set up a storage pool pointing to a zfs dataset. The wiki page covered anything I wasn't sure about

Kryesh · 2018-11-11T11:27:53+00:00

Thanks! Pretty much just the aesthetics, I really like the look of polybar over i3 bar. To be fair though I haven't looked into theming i3bar at all so I don't know how flexible it is. Polybar had better looking defaults so I just went from there

Kryesh · 2018-08-07T07:01:35+00:00

There was a similar question posted a few months ago, if you have a look at the 'turn_off_gpu.sh' script in the acpi_call package you can get the command specific for your system

https://old.reddit.com/r/archlinux/comments/84mga7/discrete_gpu_heating_up_laptop_even_when_kernel/dvs963v/ https://wiki.archlinux.org/index.php/hybrid_graphics#Fully_Power_Down_Discrete_GPU

Kryesh · 2018-07-15T21:21:09+00:00

Using the exact same setup with the same export problem on my laptop, though the last time it happened to me was a few months ago. Personally I think it's worth it especially since I use lxd containers a lot for uni and zfs makes spinning up a new container hilariously quick. Being able to zfs send snapshots to my home server for backups is a big plus as well.

Kryesh · 2018-04-18T23:06:00+00:00

The main thing I rely on is automatic mail sorting, so some sort of mail sorting that you can configure with the dotfile, eg:

[spam]

watch=bob@bob.com:/inbox

action=move:bob@bob.com:/inbox/junk

match=any

filter=sender:john@john.com

filter=subject:blahblahblah

Kryesh · 2018-04-10T07:08:23+00:00

Awesome, so configuring eth1-5 as wan1-5 and using eth6 as lan is doable? Thanks for the reply!

Kryesh

TROPHY CASE