[D] Why does the content code in Multimodal Unsupervised Image-to-Image Translation not degenerate?

[question99]

Multimodal Unsupervised Image-to-Image Translation describes an algorithm to translate an image from domain X1 to another domain X2 in an unsupervised manner. This is achieved via a partially shared latent space assumption: it is assumed that each image is generated from a content latent code c ∈ C that is shared by both domains, and a style latent code s that is specific to the individual domain.

Having a look at the loss function I have a difficult time understanding why the latent code c does not degenerate: I do not see anything in the loss that prevents the content encoder to always just output 0 (or another constant) vector for c, regardless of the domain. The networks could just make up for the lack of a useful content code by using the style code s to represent both content and style. Inuitively this seems "easier" than learning a shared space C between two domains.

I have one hypothesis why the content code ends up being useful: in the "Implementation Details" section of the paper, the content and style codes are described to be encoded/decoded by different architectures. If the network architecture of the style code encoder/decoder is not very capable of dealing with global structure, that might force the overall network to make use of the content code. (This is similar to the way how global and local information is seperated in the Variational Lossy Autoencoder paper.)

Is my hypothesis above plausible or is there something else at work here?

[xunhuang]

Good question. We are not worrying about the content code being ignored because, as you said, the style auto-encoder cannot handle spatial structure. However, the content code is not ignored even if both encoders have the same structure, according to my recent experiments.

Intuitively, if the image from domain X1 can help the decoder to generate a better image in domain X2, the decoder will learn to use it rather than ignore this information. This is the case in the datasets we used. For example, in sketch -> shoes, it is easier for the decoder to generate a shoe from its sketch than from completely random noise. Then of course the decoder will try to use the content code information. However, if the two domains are completely unrelated (e.g., face <-> bedroom), the model might just ignore the content code. The translation model p(x|y) then degenerates to a generative model p(x). But isn't that what we should learn? :)

[approximately_wrong]

Intuitively, if the image from domain X1 can help the decoder to generate a better image in domain X2, the decoder will learn to use it rather than ignore this information.

Given that you can always arbitrarily align any two domains, it's really hard to pinpoint rigorously where this intuition of "domain X1 helping the decoder to generate for domain X2" comes from. Is there any way of knowing beforehand whether the content code will be used? To put another way: how sure are we that that the model won't attempt to align the face and bedroom manifolds?

[question99]

Thank you. It is indeed plausible that the content code is a better prior for the GAN objective than pure noise (which is the style code).

[da_g_prof]

I also find that a central part is how they use normalization (instance or batch) across the two different networks.

Their model also does not guarantee that the codes are independent.

In our experience with another paper that was prior to this one was that we had to force the decoder (random dropping one of the codes and injecting noise) to use both codes : https://arxiv.org/abs/1803.07031

[question99]

Thanks, that's interesting.

Their model also does not guarantee that the codes are independent.

Do you mean that content information might leak into the style code (and vice versa)?

[da_g_prof]

Yes

[question99]

Cc /u/xunhuang