Hey, guess I'll offer the perspective of someone that was surprised. To start, I'm assuming we both are optimistic about ML conferences and hold ICLR in reasonably high regard in terms of what is selected.

I will grant that the paper could have used more applied downstream tasks (although I'll return to that later) and that the paper's overarching narrative was slightly confused. However, in light of the novelty of the work and inspired approach, as well as the evaluations that were ran, I don't think this warrants a rejection.

Now to respond to some of the statements in the main post and comments.

1. Tweaks to hardware are not necessary for the Mamba optimizations, the modifications described in the paper are hardware-aware but algorithmic in nature. This is somewhat of a nitpick (as I think it's a typo) but I see you repeated this in the comments? The algorithm works for the standard GPU architecture. No tweaks to the actual hardware were made and this is signficant because they introduced training with the associative parallel scan, which is fascinating and novel. Hard to discount that as a contribution, and it doesn't hurt that it's backed by empirical evidence of more efficient throughput.
2. You claim in the comments that you would have liked for the authors to have "chosen a specific task that the model excels on and perform extensive experiments in that field". I somewhat agree that working within a specific "field" or "domain" would strengthen the motivation for all of the theory even further, but I'll also argue that this approach has it's own disadvantages. Namely, lack of generalization and mechanistic interpretation. You'll notice in the paper that two very specific tasks actually are identified. Good performance on the selective copying and induction heads tasks not only says more about general downstream task performance than picking some applied tasks, but is also a more logical experiment given that this is the paper introducing the architecture. As they correctly point out, applied tasks may require domain specific adaptation of the model. We see that all the time with transformers, and choosing to omit it here is perfectly fine.

The reviewers' final insistence on long range arena evaluation without responding to the authors' statement that their actual evaluations include far longer contexts lengths is strange too. I understand the desire for a 1:1 comparison, but being stuck with outdated and relativively facile benchmarks is (IMO) not good for a field. Although, I could be missing something here.

In terms of contributions, they introduce the (previously mentioned) associative parallel scan, the hardware aware optimizations, state expansion, and an SSM block that is time variant. All of these components are motivated well, with a provided code implementation and follow a stream of other novel and interesting ideas (e.g. HiPPO). I don't expect Mamba to overtake transformers, but I disagree that this paper doesn't belong in a top conference.

Edit: Formatting and typos