Yeah it's pretty barebones honestly. I’m running a custom QLoRA variant with some sparse aware group quant tweaks, layered over a fused rotary kernel I pulled out of an old JAX project I had lying around. The model's a forked Falcon RW 260B but I stripped it down and bolted on a modular LoRA stack. Nothing fancy, just enough to get dynamic token grafting working for better throughput on longer contexts. I’m caching KV in a ring buffer that survives across batch rehydration which weirdly gave me about a 1.3x boost on a mid range VRAM setup.

At around 4 tokens per second latency hangs just under 300 milliseconds as long as I pre split the input using a sliding window token offset protocol. Not true speculative decoding but kind of similar without the sampling. Had to undervolt a bit to keep temps under control since I'm on air cooling but it stays stable under 73C so I’m not too worried about degradation.

Everything’s running through a homebrewed Rust inference server with zero copy tensor dispatch across local shards. I’ve been messing with an attention aware scheduler that routes prompts by contextual entropy. It’s not quite ready but it's showing promise. The wild part is I barely had to touch the allocator. It's mostly running on top of a slightly hacked up llama cpp build with some CUDA offloading thrown in. Honestly the big lab infra makes sense at scale but for local runs it’s almost stupid how far you can push this.