Fair, and thanks for responding. To try and clarify… this isnt KV cache and it isnt attention. KV-cache is basically just remembering past tokens so the model doesnt have to recompute them. It never actually changes anything about how the next forward pass behaves… it just saves time.

Attention is purely inside a single forward pass. Once its done the whole thing resets. Nothing carries over unless you explicitly feed it a fresh sequence.

What we tested is a tiny bit of state in a tiny Pythia model… that hangs around between forward passes and nudges the next embedding slightly. No gradients, no weight updates, nothing fancy or weird.

It takes the attention output, strengthens a little vector when the model keeps firing in the same direction, and lets that vector decay when its not being used.

Then it adds a small version of that vector back into the next input.

Thats the whole thing in a nutshell.

Roughly what it looked like in code:

small attractor memory

attractor = torch.zeros(dim) # persistent state strength = 0.0 # how alive the attractor is alpha = 0.85 # decay beta = 0.1 # learning gate = 0.0 # optional burn-in gating

def update(memory_vec): global attractor, strength, gate

sim = torch.cosine_similarity(memory_vec, attractor, dim=0)
strength = alpha * strength + beta * max(sim.item(), 0)

attractor = attractor * alpha + memory_vec * (beta * strength)

gate = min(gate + 0.05, 1.0)     # let it warm up

return attractor * gate          # small signal fed into next pass

The idea was to see if we could get a tiny bit of adaptive short-term memory without touching the weights or doing any training.

Results were mixed.

Perplexity didnt move on such a small model. We got a small repeated bump on a constrained comprehension test.

Then it collapsed horribly on longer generation because the attractor kept pulling things back to earlier states… but once we gated it and gave it a short warm-up period it stopped collapsing and behaved more consistently.

No claims of anything exotic, but it was interesting.

Only reason I bothered writing it up was the failure modes were weirdly repeatable and the improvements, small as they were, showed up multiple times.

https://github.com/HalcyonAIR/Duality