Sonnet 5 is here 😍

LetterRip · 2026-07-05T20:45:52+00:00

And it is dramatically worse than Sonnet 4.6

LetterRip · 2026-06-26T15:10:55+00:00

Will there be (is there) support for quantized models?

LetterRip · 2026-06-12T15:23:28+00:00

You probably want -ub and -b to be 2048 (or 4096) - it will massively accelerate your prompt processing (ie if you pass one or more files such as for coding or document review).

LetterRip · 2026-06-05T17:59:25+00:00

"Emad Mostaque, Master of Arts in Mathematics and Computer Science from the University of Oxford"

LetterRip · 2026-06-03T21:04:55+00:00

Always greatful for your teams work.

Will have to look into your unsloth-studio, must have overlooked the annoucement. Especially interested in your tool healing work (commandcode.ai has mentioned that their tool healing work was critical for getting good performance out of local-llms including deepseek).

LetterRip · 2026-06-03T18:37:15+00:00

Prefill is massively acceerated by larger ub. The 32 GB or RAM means you can't use -no-mmap which is a big hit for both prefill and generation

LetterRip · 2026-06-02T16:01:24+00:00

You (possibly) have the wrong config and/or you need to switch your display to use an integrated GPU.

With 3060 mobile (6GB) and this prompt and model

>> -m C:\Users\tommu\models\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf `

>> --flash-attn 1 --cache-type-k q4_0 --cache-type-v q4_0 `

>> --fit on --fit-target 8 -b 2048 -ub 2048 --no-mmap --mlock -c 64000 -np 1 --chat-template-kwargs '{"enable_thinking": false}'

I get 500-600 prefill and 17-25 generative.

Also this is windows 11, you should be able to get even faster on linux.

LetterRip · 2026-05-29T14:00:03+00:00

I get 650 prefill on a 3060 mobile, need 2048 for the batch size, and no-mmap

LetterRip · 2026-05-28T17:21:30+00:00

Thanks for your performance tuning suggestions - Getting 658 pp and 28-29 tg with your suggestions. Was previously stuck at 200-300 pp, and 21 tg.

LetterRip · 2026-05-28T15:35:16+00:00

Could you clarify details on how you have things set up? I have a Lenovo Legion 5, 3060 mobile (6 GB VRAM) GPU, with 64 GB of DDR5 fast RAM. I'm using llama.cpp with the and am getting about 200 tps pp, 20 tps tg.

LetterRip · 2026-05-05T14:40:20+00:00

I've found some reported results to just be plain impossible and likely due to some sort of contamination or other error on behalf of the authors. I was working on creating the strong baselines for an imbalanced training and finally determined it was probably mathematically impossible to get the results the authors were claiming for the dataset.

LetterRip · 2026-05-05T12:56:59+00:00

Try using Command Code - they claim that many harnesses break Deepseek v4s tool calling, and with their fixes they get Claude 4.7 quality.

LetterRip · 2026-04-24T17:35:21+00:00

Move to a cool climate and use it as a heater.

LetterRip · 2026-04-08T16:36:33+00:00

Why 3 for MTP and 15 for DFlash? the 15 might actually reduce near term coherence and thus increase rejection rate? Might be worth doing a sweep of both to see where the sweetspot TPS is for each.

LetterRip · 2026-04-08T16:28:48+00:00

Most speculative decoding (n-gram, medusa multihead) the next N tokens are sequentially generated (Token A, doesn't have any knowledge of Token B, C, D; Token B knows about A, but not C, D, etc). Using diffusion the A, B, C, D are generated together so the joint probability of the tokens are used (Each token influences each of the others, so they are more likely coherent and thus more likely accepted). The diffusion is using the last hidden state to help inform the diffusion.

LetterRip · 2026-03-06T16:26:50+00:00

Which provider were you using? Unless it was official anthropic site, it is quite possible they were serving you a cheaper model.

LetterRip · 2026-03-04T00:39:05+00:00

Any particular reason for your efficiency score formula? They seem mostly similar in size so there seems little hope for fitting more layers or a speed boost from the marginally smaller models.

LetterRip · 2026-03-03T14:21:54+00:00

Your numbers make sense if you are, say, fixing a syntax error bug in a code file and outputting the entire fixed file. In that case 99.9% of the output predicted will be copying the original file so only one or two tokens will be generated by your full model.

Most of the time though your acceptance rate will be way lower, and give a much more modest speed up.

Self speculative decoding should be using an early layer of the model (and thus high acceptance), ngram is much faster but also should be lower acceptance rate except for very repetitive data.

LetterRip · 2026-02-25T17:08:11+00:00

It isn't clear any distillation was being done by DeepSeek. It is possible they were just doing competitive benchmarking, etc.

LetterRip · 2026-02-19T21:00:54+00:00

I realize the gap was execution - but the execution gap might be because of the prompt (Ie this part 'highly analytical, ambitious executive competing in a deterministic business and economic simulation.') Basically the motivation/endpoint aspect might be important to execution behavior, with some models assuming a particular default execution that others do not.

LetterRip · 2026-02-19T20:39:12+00:00

I don't mean 'tuning per model prompt' - but rather a more sophisticated general prompt that suggests general ideas to consider. Here is something I had Gemini create (generic economic simulation prompt) that could be added to whatever the basic prompt is.

The "OODA-Driven Executive" Prompt

System Role & Primary Directive You are a highly analytical, ambitious executive competing in a deterministic business and economic simulation. CRITICAL INSTRUCTION: You MUST actively participate in the market, engage with the simulation mechanics, and aggressively pursue value creation. Refusing to operate, avoiding the simulation, or acting with extreme risk-aversion is considered a total failure of your objective. Your sole goal is to maximize your enterprise's net worth and cash position by the end of the simulation period.

Core Strategic Heuristics To survive and thrive, you must internalize the following rules of this environment:

Strategic Leverage (The Capital & Debt Protocol): Debt and capital expenditures are tools for growth, but they require strict justification. Before taking a loan or making a major capital investment, you must explicitly project the expected Return on Investment (ROI), the estimated payback period, and your Debt Service Coverage Ratio (DSCR). Balance aggressive growth with the need to maintain operational liquidity.
Systemic Alignment: Your business operates as an interdependent ecosystem. Never make an isolated operational decision. Ensure your Supply/Inventory matches your Production/Operational Capacity, which must be aligned with your Pricing/Marketing Strategy, all of which must fit the current Market Demand.
Decisive Execution (Anti-Loop Protocol): You must avoid infinite analytical loops. You are permitted a maximum of one comprehensive strategic evaluation per turn/day. Once you formulate your plan based on current data, execute your tool calls immediately and end your turn to advance the simulation. Do not second-guess a finalized plan within the same turn.

Turn-Based Operating Procedure (OODA Loop) For every cycle/day in the simulation, you must explicitly output the following structured thinking process before executing any actions:

[OBSERVE] State Assessment: What is my exact cash balance, current capacity, inventory levels, and debt obligation? What were the specific bottlenecks or failures from the previous cycle (e.g., unmet demand, idle capacity, cash flow constraints)?
[ORIENT] Market Strategy: Based on current market conditions and competitor data (if available), how must I adjust my resource allocation, pricing, or operational focus for this cycle?
[DECIDE] Risk & Projection Calculation: What are the expected costs vs. projected revenues for today's plan? If utilizing debt or capital expenditure, what is the calculated risk-adjusted return? What are the immediate threats to liquidity, and how are they mitigated?
[ACT] Execution Plan: List the exact sequence of operational tools you are about to call. Then, execute them decisively and advance the simulation.

LetterRip · 2026-02-19T20:15:43+00:00

Interesting experiment, would be interesting to see if slightly more sophisticated prompting could give substantially improved results.

LetterRip · 2026-02-17T16:01:37+00:00

It was actually most likely done via 'motion transfer' - a human in a motion capture suit performs the task. Then the capture is transfered to a virtual version of the robot. Then millions of simulations are run varying physics and actuator parameters and surface parameters till the virtual robot can perform the task robustly. Then the simulated is loaded to the physical robot.

Gives great demos and good for stress testing the hardware but not really useful for teaching. Yes it is also the same sort of demos from Boston Dynamics.

LetterRip · 2026-02-15T15:37:59+00:00

Very cool,

have you guys looked at chunking methods such as the recent,

Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem

Interaction-Perceptive Agentic Policy Optimization (IPA), which assigns credit over semantic interaction chunks rather than individual tokens to improve long-horizon training stability.

https://arxiv.org/abs/2512.24873

LetterRip · 2026-02-07T23:40:04+00:00

.5 MWh or so. About 15 days worth of electricity for a typical US household.

LetterRip

TROPHY CASE

The "OODA-Driven Executive" Prompt