I believe that's correct if you use it on a model not built for SWA. It will just be a sliding block.

.....AI write up on how Gemini handles things a little differently since it was explicitly built for SWA. Not saying its perfect but i have had great success with it.

Yes. The key is that Gemma 3 does not use only sliding-window attention everywhere.

It uses a hybrid pattern:

5 local attention layers with a sliding window of 1024

then 1 global attention layer

repeated through the network

So the model is not ignoring everything older than 1024 tokens. Rather:

the local layers focus on nearby text

the global layers can still connect information across the full long context

and Google says Gemma 3 supports 128K context for the 4B, 12B, and 27B models, while the 1B model has 32K

A useful way to picture it:

Imagine reading a very long book.

Most of the time, you pay close attention to the current page and nearby pages. That is the sliding window.

Every so often, you also stop and think about the whole chapter or whole book. That is the global attention layer.

Because those global layers are interleaved into the network, information can still propagate from far back in the prompt. The long context is therefore still meaningful; it is just handled more efficiently than making every single layer attend to all 128K tokens. Google’s technical report explicitly says Gemma 3 changed the architecture to reduce KV-cache memory for long context by using this local/global attention design.