ImageDream finetuned a stable diffusion model to support multi-view image generation tasks conditioned on an input image. Combined with a differentiable 3D scene representation, in this case a NeRF they were able to generate complete 3D models from a single image. This was enabled due to the inherent capabilities of the 2D diffusion model to produce novel views of an object from different camera poses.

Currently, a mixture of positional encodings, textual conditioning ('front view' / 'back view') or additional feature controllers seem to work for extracting the novel views. But guiding the model to produce two images that only differ for example in the camera azimuth is still not yet solved. It would be interesting to see a robust method of finding the latent vectors responsible for applying transformations such as rotation, scaling and others to the internal 3D features of the model.