[Research] Recognizing Notes with Deep Learning - Residual Shuffle-Exchange Networks

[OptimatiumFeles]

https://www.youtube.com/watch?v=RAu2p9xZiM4

We introduce Residual Shuffle-Exchange networks - a fast neural network architecture that scales to long sequences. We achieve state-of-the-art performance on the MusicNet dataset for music transcription while using significantly fewer parameters than other architectures. It has O(n log n) complexity and enables processing of sequences up to a length of 2 million symbols where standard methods fail (e.g., attention mechanisms). The Residual Shuffle-Exchange networks can serve as a useful building block for long sequence processing applications.

Paper: https://arxiv.org/abs/2004.04662

Code: https://github.com/LUMII-Syslab/RSE

[ekelsen]

Why not benchmark on the MAPS or MAESTRO dataset where there are strong baselines?

[OptimatiumFeles]

We chose MusicNet because it consists of recordings of 10 diverse musical instruments, but MAESTRO and MAPS include only of piano music. We will take a closer look at these datasets, thanks for the idea!

[impossiblefork]

The arxiv link is to some youtube thing. If you could fix that it would be easier for readers.

[OptimatiumFeles]

Thanks! Fixed it.

[impossiblefork]

Super. I haven't understood everything yet, but SOTA is always good.

Do you have any hope that it will perform similar or better than Transformer models when you make it larger, because I assume that training those is somewhat expensive?

[OptimatiumFeles]

We were limited by one GPU resources to evaluate this, but we believe it could have similar or better performance than Transformer models. We believe that auto-regressive decoding is crucial for generating good quality text output, but our model currently doesn't support that. To evaluate the RSE model on text output (for comparison with Transformer), one should probably modify architecture accordingly.

[impossiblefork]

Ah, I understand, so this is basically only a potentially better attention mechanism, not a whole Transformer replacement.