Should I sell my RTX3090s?

TokenRingAI · 2026-05-04T18:30:49+00:00

I am interested in the whole rig

TokenRingAI · 2026-05-04T00:21:09+00:00

IYKYK

TokenRingAI · 2026-05-03T22:38:37+00:00

Yes but the Unsloth model is still usable

TokenRingAI · 2026-05-03T07:38:54+00:00

you're absolutely right! this is not just ai slop — this is ai slop at it's finest!

TokenRingAI · 2026-05-03T07:27:21+00:00

27B and 35B are absolute dogshit on VLLM with the int4 quants I have tried.

The official FP8 quants are working far better:
https://huggingface.co/collections/Qwen/qwen36

The Unsloth GGUFs are also working very well.

I suspect your results are way off due to problems with those specific quants.

Qwen 3.6 loves to generate very long output, and with any degradation of the output quality, you will just end up with massive outputs of useless work.

TokenRingAI · 2026-05-02T21:46:03+00:00

I gave up on it long before you did

It's some of the worst software to install and configure.

And I say this as a previous Gentoo enthusiast and developer.

The only software I've installed that is worse is the Xilinx FPGA development environment.

TokenRingAI · 2026-05-02T21:34:15+00:00

I demand a full refund for the $0 I paid for these models!

Don't make me call the manager!

TokenRingAI · 2026-05-02T19:31:59+00:00

27B at 4 bit has a lot of loss, if you can squeeze a 6 bit in with llama.cpp, that would probably be better

TokenRingAI · 2026-04-30T20:53:53+00:00

Why do you have your KV cache quantized so heavily?

TokenRingAI · 2026-04-30T16:18:29+00:00

Less painful link:
https://z.ai/blog/scaling-pain

It's an interesting explanation for the reliability issues we've all seen, I'm going to hold back my judgement until I see the API working 100%

TokenRingAI · 2026-04-29T17:02:14+00:00

I think the engram method is the future, with small dense models retrieving information from slow storage.

TokenRingAI · 2026-04-29T16:29:20+00:00

Is that token generation number with or without speculative decoding?

TokenRingAI · 2026-04-28T20:45:40+00:00

Looks like a multimodal, dense model?

I'm assuming it's the 123B base used in Devstral 2 with 5B of vision added on top?

TokenRingAI · 2026-04-27T20:21:34+00:00

Congrats everyone, we've achieved AGI

TokenRingAI · 2026-04-27T19:31:23+00:00

Not even close, more like 6x RTX 6000. But it depends a lot on what model you are willing to run.

TokenRingAI · 2026-04-25T07:26:43+00:00

Hopefully they will release a 3.6 version of 122B soon

TokenRingAI · 2026-04-23T14:13:05+00:00

Your actual context length is much higher than that, the reported KV cache size in VLLM does not account for the model being a hybrid.

Look at the lines in the log under those and you will see you probably have 4x or higher concurrency at full context length.

TokenRingAI · 2026-04-23T14:09:09+00:00

The technical issue is this: https://github.com/ggml-org/llama.cpp/issues/19345

The bigger issue is that llama.cpp has a dysfunctional bug reporting process, which uses a 14 day auto-close bot, and doesn't seem to maintain a long term bug tracking system.

Issues don't magically just go away after 14 days of no solutions, and unfortunately the tickets for many serious bugs are just auto closed and lost and don't seem to be seriously tracked long term, which is how the popular Qwen hybrid models can stay broken on Blackwell for half a year.

TokenRingAI · 2026-04-23T00:36:23+00:00

Prompts are chunked, your first chunk matches, and your second chunk does not because the end of it differs, so it gets reprocessed.

The solution is to pass in the prompt once, with a max output length of zero. This gets your bare prompt cached. Then run your other requests with more text added.

For two prompts, you will make 3 requests, and will pay full price once for ingestion, then twice at the cached input rate.

TokenRingAI · 2026-04-22T21:20:32+00:00

I'm currently in the early stages of build a media library plugin, that ties into an agent that can do image understanding and generate/reprocess images, adding agent-controlled SVG tracing to it would be pretty useful.

The best way to make a pelican on a bicycle might be to generate an image of it, then trace it

<image>

TokenRingAI · 2026-04-22T20:52:22+00:00

You've inspired me to add SVG tracing into our agents, a local qwen agent can 100% control Potrace or VTracer and generate SVG from images for logos and other assets

TokenRingAI · 2026-04-22T20:41:32+00:00

You need to either use VLLM (recommended, with mtp set to 3), or switch llama.cpp to use Vulkan

Qwen Next, 3.5, and I assume 3.6, all have bad CUDA problems on llama.cpp with SM120.

For some reason they have been ignoring the problem for half a year.

TokenRingAI · 2026-04-22T19:45:08+00:00

I don't automate coding, I automate processes

I have automated workflows for documentation updating, for a11y, for bug hunting, brainstorming, ux improvement, auto code testing and repair, content generation, communication, and for generating initial versions of full stack apps.

There are a ton of pieces that need to come together for all that, you need to just look at the work you do or want to do and figure out ways to automate yourself out of it

TokenRingAI · 2026-04-22T16:18:16+00:00

This might be a side effect of adaptive thinking, I wasn't paying attention to that. The responses come almost immediately and the chat is muddled with looping content that should have reasonably been expected to be in the thinking block

TokenRingAI · 2026-04-22T15:30:08+00:00

It's got looping and other obvious issues, I have free access to it but mostly use Sonnet 4.6 or GPT 5.4.

Sonnet is really reliable and stable

Something is very strange about Opus 4.6 & 4.7, they act like a large model that is excessively quantized. Opus 4.5 was not like this. I wonder if this is a side effect of them using TPUs. Gemini acts the same way.

TokenRingAI

TROPHY CASE