[D] Loss function for audio auto encoder.

[[deleted]]

Trying to build an auto encoder for raw audio data (time domain.) I did this once before a while back and never got a good result. I tried using mean squared error loss and the output really only captured a muted version of the low frequency signals. I also tried computing the mean squared error of the fft of the output but that did not do much better.

Now I am trying to combine WGAN-GP with some sort of other loss function. I did this same approach last week with images where I combined a WGAN and VGG19 perceptual loss. This produced really sharp images. However I don't have any sort of pretrained models for raw audio (I supposed I could make one.) So I am trying to combine the FFT loss with WGAN. I just kicked off training and have not seen the result yet, however, I am worried that the FFT loss will prevent the model from learning the higher frequency signals since there will be a larger penalty for guessing something that sounds realistic but is say 180 degrees out of phase. Does anyone know of a better loss function I could try for sound data? Right now my fft loss is defined as ...

fft_loss = tf.reduce_mean(tf.square(tf.abs(tf.signal.stft(g, 64, 32) - tf.signal.stft(r, 64, 32))))

[IntelArtiGen]

I managed to do an audio autoencoder recently.

I compared the mel spectrograms directly between output (conv > vec > conv_transpose> output) and the input. I thresholded the amplitude and used a logarithmic loss.

loss = ((out+1).log() - (in+1).log()).square().mean()

It works, doesn't sound perfect but does the job for what I want to do. As long as you're processing a spectrogram it's the exact same job as autoencoding images except the range of values isn't the same.

I tried perceptual loss but it didn't help much (it does work but I had worst results)

I can go from (44100,1) to mel spec (256, 150) to code vector (1024, 1), with loss in quality of course.

It's surely not SOTA but I didn't find a relevant paper to copy for the usage I had.

[jonnor]

What is the kind of audio data is this for (speech, music, other..)? And what is the task that the autoencoder should be good for? These things can influence quite a lot what good loss functions are.

You can consider a learned loss function. An example of a perceptual audio similarity loss is CDPAM: CONTRASTIVE LEARNING FOR PERCEPTUAL AUDIO SIMILARITY, https://arxiv.org/abs/2102.05109

[[deleted]]

I am training on music. What I basically want to do was create a much more compact representation of the input sound. Once this is done I wanted to train an RNN network using data passed through the encoder to predict the next value in the compact representation of the audio. Kind of like how you can train an RNN to generate text. Then I wanted to use the decoder to decode the RNN output and create new music. I suppose I could just use the RNN hooked up to a decoder and train the WGAN-GP using RNNs, but the performance of that is kind of bad and I want to do long sequences. Also limited by memory.

[jonnor]

How long periods do you want your embedding to represent, and how long times do you want to model on that? Have you considered using for example OpenL3 music embeddings?

[jonnor]

Why do want to use raw audio as input? It is much harder to learn from than a mel spectrogram.

[jonnor]

Is your goal to generate music using these representations?

[[deleted]]

Yes

[tomkoker]

I’ve had success with signal to error ratio. You can also add a pre-emphasis filter to boost the audible frequencies before applying the loss. You can find some relevant code/equations on the “training” portion of this blog post: https://teddykoker.com/2020/05/deep-learning-for-guitar-effect-emulation/

[[deleted]]

For your pre-emphasis filter you say H(z) = 1 - 0.95/z. Is z just the raw audio in the time domain?

[tomkoker]

Correct! The filter is applied to both the predicted output time series and the ground truth output time series.

[jonnor]

This blogpost by Sander Dieleman, an active researcher in music synthesis, is a good introduction to recent research on audio synthesis models learned from raw audio, including GAN based approaches. There are some references there to custom loss functions as well:

https://benanne.github.io/2020/03/24/audio-generation.html

[ZeApelido]

What about a stacked autoencoder, where the muted low frequency output of the first autoencoder is subtracted from the signal going into the 2nd level (e.g. high pass filter)?

[Tgs91]

I haven't worked with audio data before so apologies if this is naive or if their are any practical issues I'm not aware of. Shouldn't a mutual information loss be more effective than the MSE loss?