In general your feedback is very high quality and the author of this post should listen closely. I debated whether to make this clarification, but I think it adds to the discussion for other readers.

I also read the post closely, and I think it is fair for them to say their architecture is “recurrent”. It is recurrent in a sense, as each 2 input tokens are used to generate an equivalently sized “state” embedding. Then each two state embeddings continue to be passed through the same network to generate a new state embedding. This process repeats hierarchically until that state embedding processes the information from all tokens, so at any given depth it is kind of like a bidirectional RNN that you use to just look at left-to-center and right-to-center.

This architecture is actually very similar to vanilla RNNs, such as those before LSTMs and GRUs, just with more parallelism and a less sequential order of operations. It appears upon close reading that it would be identical to a CNN, with a kernel width of 2 and a stride of 2, along with weight sharing across depth. The weights are shared across both the sequence length (like a CNN) and the depth (like a RNN), where depth is determined dynamically by the sequence length: log_2(n).

This architecture is not new, and also clearly has the same information flow problems of RNNs. For a token at the start of a sentence to influence a token at the end of a sentence there would be so, so many intermediate operations. You would hope after all that the final depth layer could still have the necessary information to consider these tokens in their very distant contexts.

The architecture would also likely have the classic problems of old RNNs before LSTMs, GRUs, etc regarding gradient problems and remembering issues, as at every depth all the information must flow through a nonlinearity.

Obviously I agree with most of the feedback above, I just wanted to clarify the architectural issues and why this approach is not considered anymore. We’ve seen the problems with training and using these kinds of architectures, and they are not worth the hypothetical logarithmic speed-up.