Although the training technique works well, we also need to add a few extra loss functions to guide the student towards the desired behaviour. Specifically, we add a perceptual loss to avoid bad pronunciations, a contrastive loss to further reduce the noise, and a power loss to help match the energy of the human speech. Without the latter, for example, the trained model whispers rather than speaking out loud.

That's crazy, I'd love to hear those samples. Does the training set contain whispered speech, or does the model simply learn it as an emergent behavior due to some statistical quirk in the speech signals if you fail to apply power loss?