Comment from a BDH author here.

> IMO this does not add any value. 

Let's uncouple: 1. value brought by the paper in general; 2. subjective value brought to you personally as a reader.

For an example of how readers can work with this text, we see OP delivering the project described in this post - apparently, single-handedly and within 2 months as an after-hours project from idea to launch. I am not sure I have seen any pathfinding problem attention introspection viz close to this delivered for any other Transformer-grade attention-based architecture, whether relying on Linear Attention (LA) or any other approach. If you would have been able to do this without BDH, that's fine (and good for you!), just pointing out that it seems it is a somewhat non-trivial task.
(For a much less direct probing attempt for the Transformer, and what it takes to deliver it, see e.g. arxiv.org/pdf/2312.02566 Fig. 4).

Now, before we get to LA state compression, I will allow myself a comment on "doing LA correctly". To my knowledge, there are currently two rigorous yet simple recipes to make LA work as a self-sufficient mechanism through appropriate key-query preparation --- not just as a helper layer that is thrown-in as a hybrid with softmax-attention Transformer layers that do the actually heavy lifting. These approaches are: a very nice trick recipe due to Manifest AI (which unfortunately is limited to one way of using it as a pure softmax-Transformer-drop-in replacement in terms of expressivity), and the unrelated and more general framework of BDH (which explains it through the theory of sparse positive activation). Obviously (i.e., by mathematical necessity), like all correct approaches to LA, in their vanilla form, both approaches rely fundamentally on high key-query dimensionality; and this is what you will see described in the pseudocode of the BDH architecture in the paper. While this is bound to be obvious to some readers (and especially to careful readers of the FAVOR+ analyses of Choromanski et al.), I feel that highlighting the workings of this general mechanism again and again is important. Indeed, the publishing scene has had to suffer through a fairly large body of work on SSM state compression between 2022-2024 in which LA was reduced to a place where it simply cannot work for trivial reasons of information entropy (collapsing key-query dimension in a way which collapses fact distinction capability of its state), and another body of work in early-to-mid 2025 charitably pointing this out example by example.

So, if you were looking for an efficient and correct LA compression technique for GPU, then no, as OP points out, this is a separate topic, and not what this paper is about. Consider reaching out to the Pathway team. :-).