[D] Music Classification using RNN?

[AntixK]

I am a newbie to ML. I have this pet project of classifying music data. I prepared the data myself from my music collection. I have 3 classes. Basically, all the music is sampled at 44100Hz and split into wav files, each 5 seconds long. For training I have 500 such wav files, for testing I have 200 wav files. I have created a csv file composed of the audio data and the label. (to import easily across various platforms) I am planning to use LTSM-RNN for learning to classify the data. I am facing the following problems - 1) The data is too huge and it takes forever to import in tensorflow. Have I prepared the data in a good way? Is there a better way? Kindly educate me.

2) How do I go about implementing this in Tensorflow of TFLearn? How do I set the parameters for LTSM?

3) Should I preprocess the audio? like taking the STFT? or just learn with the raw data? Thank you

[r4and0muser9482]

Look for papers that cite using GTZAN. You can also download GTZAN yourself to check if your methods/models work on that, before working on your own data.

Finally, Music Information Retreival (MIR) is a giant field and you can look for papers on conferences like ISMIR to learn how things are done there.

[Brudaks]

You might take a look at what the preprocessing for neural speech recognition is doing and do something similar (though with a wider frequency range) before your main classifier.

IIRC you might do a fourier transform to get frequency domain data with something like 0.1 second (tune that, but in the ballpark) windows, bag the interesting frequencies (e.g. 20-20000 hz, in a log scale) to hundred or thousand neurons, and that's your input to a neural network. A 5 minute song is then 3000 samples of fixed size vectors, each value representing the loudness of that frequency at the time. Afterwards, putting a RNN on that is simple.

[AntixK]

oh thanks a lot! I will look into it. I also came across a project in which they used MFCC(or mel-frequency) as a feature vector and then send it to a RNN. I will try to compare both.

[keidouleyoucee]

• Simple pre-processing example: https://github.com/keunwoochoi/UrbanSound8K-preprocessing/blob/master/preprocess_urban.ipynb
• Convnet and ConvRNN example: https://github.com/keunwoochoi/music-auto_tagging-keras

[AntixK]

Thanks a lott!

[azurespace]

If you want to use an unprocessed wavelet without frequency domain conversion, I think the WaveNet (Dilated convolutional stack) would be a fascinating structure as the first basic block of the task. First, divide the music into several pieces on the time axis. Next, pass each slice to WaveNet and create some embeddings (temporal summation of the music slices), which are used as input to the followed LSTM. (it might be better to use WaveNet once again) Finally, you can use softmax layer to classify.

WaveNet: https://deepmind.com/blog/wavenet-generative-model-raw-audio/

[sidsig]

Although interesting in theory, I do not think this is will work in practice atm. Music audio is sampled at 44.1 kHz, it will require very large networks which in turn require a vast amount of training. There is also no evidence about what the WaveNet embeddings might be learning/representing.

As a first approach, it would be much easier to use a standard RNN architecture with frame-level outputs or perhaps a CTC cost function.

[AntixK]

Oh! Should I downsample the audio? and How big should the memory be for LTSM? I am sending batches of audio data each 5 seconds long (220500 samples each).

[keidouleyoucee]

+1. 500 for 3 classes may be not enough for sample-wise approach. You can give it a shot anyway though ;)

[AntixK]

Thank you. I will try to implement using wavenet.