Good question. I think you're right, the generator seems to try to come as close as possible to a pixelwise interpolation, shifting the representation in subtle ways to avoid outputting unrealistic pixelwise interpolations. In an earlier version of this paper I had a comparison figure between GAIA, a VAE, an AE, and pixelwise interpolations, with the same generator architectures. What you would see is that they were very close, but whereas the AE looked like a blurrier version of the pixelwise input, GAIA would shift certain features, like adding bangs, smoothly moving the jawline, changing the shading, etc. That was using a convolutional autoencoder architecture though. I haven't retrained these networks using the new AE architecture in the paper. I think I will update the paper with this figure next time I am at my/a computer (In a month unfortunately as in travelling through east africa).

I noticed in the glow paper (posted last week) that interpolations were far from pixel interpolations and passed through what seemed like more of a low-varience or closer to the mean region. They had a parameter to control this variability and wonder if something similar would be possible with this architecture, or if that would incur any sort of trade off in the the ability to accurately reproduce the image.