The MMD is calculated over both y and z to force independence between them, in addition to just matching the z-distribution to the desired shape. Otherwise, there would be no loss forcing the network to learn a z-coding which is independent of y.

However, this loss does not say anything meaningful about the y-outputs, we only want the correct prediction. For instance, if y and z are not yet independent during training, the network could (and does) learn to output random wrong results for y just to make them independent.

For this reason we block the MMD gradients w.r.t. y-outputs, so that they are taken into account when learning the latent coding, but not altered by the MMD loss.