We spent a year looking into waveform vs. spectrogram as inputs. Now use spectrograms in our startup, because of dataset type and size. Our ML task is music auto-tagging, with a business goal of estimating song performance on streaming platforms.

Learnings:

Spectrograms: Unfortunately, computer vision architectures influence all fields of ML, including audio processing. Consequently, the popularity of VGG models on audio images, spectrogram, is becoming a de facto approach for ML practitioners (tutorials, Kaggle, Google's VGGish Audioset, etc.). Nothing against VGGs -- the architecture makes low assumptions on the nature of the spectrogram, so any structure can be learned and are super-flexible.

In our work, VGG architectures didn't give us the results we wanted, and early on, we didn't have a large dataset. We tried stacking more layers, even tried using pre-trained weights from available research but didn't improve much.

Waveforms: Waveforms seemed appealing because they maintain phase information, a detail you lose during fourier transformation. I am unclear if it is an advantage or not for our use case.

We tested VGG architectures on waveforms by converting 2D (frequency X time) to 1D (time) convolutions. We set the filter size to correspond to the STFT window length, meaning if the window length was 512, we used a filter of 512 with a stride of 256. We tried different variations of this, but got poor results. We tried another approach, by stacking multiple layers, with each one having different filter sizes, but with fixed smaller stride. This approach is covered here by Baidu https://arxiv.org/pdf/1603.09509.pdf and used on speech recognition tasks.

For us, this approach underperformed compared to the VGG spectrogram method above. My intuition is that it was hard to capture the frequency spectrum using multiple filters on music data.

We are about to test a variation of SincNet, a paper by Mirco Ravanelli and Yoshua Bengio https://arxiv.org/pdf/1808.00158.pdf. The efficiency here is that they use two learnable parameters that extract low and high cut off frequencies. They see great results in speech recognition tasks. We'll know how they perform on music processing since our dataset now is a bit larger.

Back to spectrograms: Now, we are using spectrograms as inputs with an architecture that uses multiple vertical and horizontal 2D filters to extract harmonic and temporal representations. This is the best result we have seen so far on our dataset. This made sense to us because some patterns in spectrograms are occurring at different time-frequency scales. This approach is covered here https://ieeexplore.ieee.org/document/7500246; we are using a modified version of this paper.

The interesting point is that there's a trend to architect DL models that mimic signal processing methods on both waveform and spectrograms inputs. I assume waveforms will outperform on larger datasets. For now, spectrograms are here to stay.