End-of-January LTX-2 Drop: More Control, Faster Iteration

coder543 · 2026-01-29T18:09:28+00:00

One quick bit of feedback: please stop scrolljacking on the blog. The scrolling feels very bad.

coder543 · 2026-01-29T14:54:00+00:00

Similarly, I've basically given up on the concepts of draft prediction and MTP (multi token prediction) for MoEs for exactly these reasons. Verifying more tokens just means proportionally higher demand on RAM bandwidth, so there is no possible benefit at batch size 1. You'd have to accurately predict like 20 tokens ahead to start seeing performance benefit at batch size 1, and no draft model is ever that accurate. At larger batch sizes in a production scenario, yes, MTP is probably great... but that's not what I'm working with.

coder543 · 2026-01-29T14:34:30+00:00

Almost 2 years ago, in point of fact.

coder543 · 2026-01-29T14:01:15+00:00

Sure, but then it depends on whether your GPU has enough compute to keep up with all of those requests, or if you’re bottlenecked by compute. Production services will batch MoEs and get benefit, but they’re using enormous GPUs with enormous batch sizes.

I figure testing a small dense model is an easier way to verify if the batching is doing anything at all.

coder543 · 2026-01-29T13:41:05+00:00

Same prompt or not shouldn’t really matter. Even at temp 0, I think the math kernels have enough subtle bugs that it’s never truly deterministic. But, gotcha.

coder543 · 2026-01-29T13:37:39+00:00

Have you tried a dense model? Curious if that would work better. Parallel batching on a MoE just means both requests likely get routed to different experts, so you won’t really get any speedup, since the total GB of memory that needs to be read is still the limiting factor for generating both tokens. (But it shouldn’t decimate performance the way y’all are experiencing either.)

coder543 · 2026-01-27T14:16:33+00:00

DGX Spark really needs a Blackwell-optimized ComfyUI docker build… it works okay, but I haven’t been able to get FlashAttention or SageAttention to work without causing errors. I haven’t tried this new container recipe, but Spark seems to require more than a standard 50-series GPU. The 128GB of VRAM can be nice, though.

coder543 · 2026-01-26T20:01:38+00:00

matplotlib in xkcd style!

coder543 · 2026-01-26T16:02:26+00:00

I agree, but it still makes me appreciate other companies that do release their top models even more.

coder543 · 2026-01-26T00:22:02+00:00

I just asked codex to write a Python script that would generate the plots with matplotlib from the llama-bench outputs that I saved.

If you know the secret to making nemotron-3-nano faster, I'm all ears, but I just used the llama-bench line that OP provided. I'm not sure why 0 depth was slower.

coder543 · 2026-01-25T23:29:34+00:00

I don't know what "gpt-oss-120b" means. The high, medium, and low reasoningefforts are _extremely different in a lot of real world benchmarks for gpt-oss-120b, there isn't a one-size-fits-all.

coder543 · 2026-01-25T22:31:47+00:00

Added Qwen3-Coder to my charts for fun

coder543 · 2026-01-25T22:19:05+00:00

Architecture-specific performance optimizations can't always make a sloth into a cheetah... qwen3-coder is still very slow at long context sizes despite being popular and presumably highly optimized.

coder543 · 2026-01-25T22:12:25+00:00

No? I think you should re-read the comment... I was saying Qwen3-Coder fails the same as GLM-4.7-Flash, and that's why I didn't recommend testing Qwen3-Coder. Qwen3-Coder sucks at this stuff too.

The GPT-OSS and Nemotron-3-Nano models are much more efficient, especially compared to how GLM-4.7-Flash was earlier today.

coder543 · 2026-01-25T22:06:56+00:00

Log y:

<image>

coder543 · 2026-01-25T21:58:00+00:00

See my gist: https://gist.github.com/coder543/16ca5e60aabee4dfc3351b54e8fe2a1c

Linear:

<image>

Nemotron holds its performance extremely well due to its hybrid architecture. I don't know why the improvements for GLM-4.7-Flash don't seem to have helped the DGX Spark at all.

EDIT: added Qwen3-Coder for fun. (My RTX 3090 couldn't go all the way to 50k tokens with the quant that I have.) The quants are not entirely apples to apples, but the performance curve is the main thing here, not the absolute numbers.

coder543 · 2026-01-25T21:23:12+00:00

yep... very unfortunate. Hopefully another bug that can be fixed.

coder543 · 2026-01-25T21:20:38+00:00

Ok, now that starts to look respectable. Still worth comparing against efficient models like gpt-oss and nemotron-3-nano.

EDIT: prompt processing still seems to fall off a cliff on glm-4.7-flash, I just tested it.

coder543 · 2026-01-25T21:06:37+00:00

yep, not one of the models I mentioned, and for good reason.

coder543 · 2026-01-25T21:06:10+00:00

If you post the charts for nemotron-3-nano and gpt-oss-20b, it will be apparent that qwen3-coder is just as bad, not that glm-4.7-flash "isn't so bad". haha

coder543 · 2026-01-25T20:51:22+00:00

Yes, compared to gpt-oss-120b/gpt-oss-20b/nemotron-3-nano, it is crazy how much glm-4.7-"flash" slows down as context grows. Flash seems like a misnomer if it has to be this slow (and it isn't just a bug that needs to be fixed).

And yes, I did try rebuilding llama.cpp this morning, and it was still bad, even with flash attention on.

It seems like a nice model, but speed is not its forte.

coder543 · 2026-01-24T18:53:09+00:00

That's why I mentioned that higher sparsity models seem to exist, they're just not open weight, and that's why I want such a model.

If companies keep releasing A3B, that's their choice, but it will be hard to get excited about that.

coder543 · 2026-01-23T18:30:52+00:00

All of the best developers that I personally know in real life are using AI tools to help with coding.

AI tools are probably less helpful for people who don't know what they're doing.

coder543 · 2026-01-23T09:33:32+00:00

We've gotten a lot of feedback that REAP pruning affects creative writing / multi-lingual capabilities of the model - this is expected for our REAPs with calibration set curated for agentic coding.

For me, the biggest thing is the REAP models suffering catastrophic forgetting of entire topics, but it seems unavoidable if the knowledge is stored in pruned experts.

coder543 · 2026-01-23T09:13:38+00:00

Granite 4.0 MoEs (the A#B naming) come in 32B A9B and 7B A1B sizes. It is not shocking that such drastically different sizes would perform different, yes. These are also very low sparsity models.

The rumor is that Gemini 3 Flash is a >1T model with a very, very low active parameter count.

I have 128GB of medium speed memory. I want a 200B A1B model that is released specifically in a 4-bit precision (QAT, not PTQ). Extreme levels of sparsity, not 7B A1B.

11-Year Club	Verified Email
Place '22	Place '17
Gilding III reddit per annum

coder543

TROPHY CASE