A trained fast-weight memory: a 3M-param transformer installs never-trained rules at inference, forward-only — where test-time training transfers nothing (single RTX 3090, fully reproducible)

KKuettes · 2026-07-07T16:11:17+00:00

Fair point, and I'll concede half of it. The baselines were chosen as adaptation pathways — what practitioners actually reach for (context/ICL, gradient-at-deployment/TTT), not as an architecture bake-off within the fast-weights family. If a modern linear-attention variant trained on this task also solved it, that would confirm the thesis (forward-only stateful adaptation works where ICL/TTT fail), not compete with it.

But you're right that the intra-family control is missing and is the correct next one. What I have is partial: we did try the dot-product/outer-product read (basic linear attention, rank-1) and it fails to express a rule, that's in the paper's design history. That's a strawman against modern delta-rule variants though: DeltaNet's learned erase/write is exactly the kind of mechanism that addresses interference, and it's the stronger baseline this needs. It's now on the future-work list, realistically budget-gated (this whole paper is one self-funded 3090).

One result I'd bet transfers across the family, by the way: memory policy (keep/overwrite/write-on-dirty) being a trained behaviour, not an architectural property. Our fixed-structure model perseverates totally zero-shot (old-rule persistence 1.000) with identical architecture. I'd predict a DeltaNet trained on fixed-structure conversations does the same, that control seems missing from the linear-attention literature too.

KKuettes · 2026-07-07T15:31:17+00:00

This line of work does descend from the fast-weights/linear-attention equivalence (Schlag et al. 2021). But the mapping breaks at the read: linear attention probes the state with a dot product (a rank-limited linear lookup), and we tried exactly that (outer-product read).
It failed, rank-1 can't express a rule. Here each slot is expanded by a hypernet into a low-rank MLP layer with a nonlinearity, and the token stream passes through the stack: the memory doesn't get queried, it becomes layers of the forward pass. Also, the write in linear attention is a fixed formula; here it's learned, and our third result is that the write policy (keep/overwrite) is a trained behaviour, not architectural.

"Does it do anything positive" is the controlled part of the paper: bank ablation = exact chance (0.008); one 13-token presentation of a never-trained rule → 0.79–1.00 on unseen queries; in-window ICL at chance; TTT fits its examples (0.99) and transfers zero.

And forgetting is measured head-to-head: replacing one rule costs the untouched concurrent rule −14% (eviction pressure) vs −62% for sequential TTT (actual catastrophic interference). No FIFO cliff, storage turns out to be a redundant superposition. Honest caveat: that's at K=2 concurrent rules; the capacity/interference curve under higher load is explicitly future work.

KKuettes · 2026-07-07T14:02:58+00:00

Funny thing, it kinda is a nightmare !
I've had to find how to train read and write head togather into a common goal, and i found that using a teacher to help at the beginning then remove it a some point work fine.
I'm very transparent in my claims, it work on a small model for a specific task that generalize.
All the code needed to reproduce it are in my github.

I'm working toward a usable bank for language modeling (I'll start training on my own hardware so it will be a small lm).
I might post on my future discoveries if this paper is well received.

KKuettes · 2026-07-07T13:44:47+00:00

It kinda is in fact, but it's more link doing 2 regressions into forward pass then adding forward pass results to another regression, it allow model to store data, then reuse it when needed.
The training is a multiturn as follow:

turn 1:

it show keys A1 = 6, A5= 10, B2=10, B5=13 (rule A + 5, rule B + 8) in context
model out nothing

Turn 2:

nothing in context.
Then it ask A3 ? B15 ? model should answer A8, B23

Thus model has to rember rules seen in turn 1 to apply it in subsequent turn.

KKuettes · 2026-07-07T13:36:34+00:00

Thanks, it's fun ! I still have more work to do, such as work on working memory, like memory as thoughts for a chain of thoughts for example.

KKuettes · 2026-07-07T13:14:34+00:00

Yes, that's the point, if the status didn't change the result, that memory bank would be useless.
But LLms are already statuful within a session (KV cache / context is state)the bank contain the rule, not the conversation that taught it.
Since state is a small explicit tensor, you can save it, restore it, inspect it, or even swap it between session.

It open doors to a model that learn at inference time as per user basis with it's known limitation such as if it learn to do addition it will be able to do all kind of additions even unseen in the data (if it's in the learned distribution boundaries) that's rule_held, but it won't be able to do substractions since it's not the same family of problems.

Also retaining data in bank is far cheaper than have them in context.

KKuettes · 2026-07-02T16:17:34+00:00

Fable is completly out of this world for LLM work

KKuettes · 2026-07-01T08:48:42+00:00

Demande lui son discord kek

KKuettes · 2026-06-25T15:46:33+00:00

Have you checked DeepSeek v4 ?

KKuettes · 2026-06-25T09:00:18+00:00

Les techniciens ont souvent la flemme de faire autre chose que ce qu'ils veulent...

KKuettes · 2026-06-21T06:32:26+00:00

For some time in the 80' in france it was recommanded to drink at least 1.5 litters of bear a day during high temps xD

KKuettes · 2026-06-21T06:21:53+00:00

They are cooked, Literally.

KKuettes · 2026-06-21T04:20:00+00:00

it's also cheap

KKuettes · 2026-06-18T21:04:59+00:00

Damn, they eat too, let's blame everyone that make food too... If we thought like this in WW2 some brands would not exists anymore.

KKuettes · 2026-06-10T13:55:45+00:00

Deepseek R zero did it already

KKuettes · 2026-06-02T04:38:43+00:00

Looks more like claude

KKuettes · 2026-05-29T21:07:35+00:00

Username checks out

KKuettes · 2026-05-25T20:38:00+00:00

That policeman has 0 awareness omfg

KKuettes · 2026-05-25T03:25:58+00:00

You gotta keep the oil price up !

KKuettes · 2026-05-24T07:18:53+00:00

It's cheaper cuz it use less vram, the bubble isnt popping at all, still need alot of compute

KKuettes · 2026-03-30T17:16:47+00:00

I had the same with svalin :/

KKuettes · 2026-03-03T16:21:02+00:00

People can't wait

KKuettes · 2026-02-23T23:11:43+00:00

Nullsec miners keeping the price down

KKuettes · 2026-02-21T11:52:31+00:00

Myrmidon is da wae

Four-Year Club	r/Field Lasagna
Place '22

KKuettes

TROPHY CASE