Modern learning-based approaches to 3D-aware image synthesis achieve high
photorealism and 3D-consistent viewpoint changes for the generated images.
Existing approaches represent instances in a shared canonical space. However,
for in-the-wild datasets a shared canonical system can be difficult to define
or might not even exist. In this work, we instead model instances in view
space, alleviating the need for posed images and learned camera distributions.
We find that in this setting, existing GAN-based methods are prone to
generating flat geometry and struggle with distribution coverage. We hence
propose WildFusion, a new approach to 3D-aware image synthesis based on latent
diffusion models (LDMs). We first train an autoencoder that infers a compressed
latent representation, which additionally captures the images' underlying 3D
structure and enables not only reconstruction but also novel view synthesis. To
learn a faithful 3D representation, we leverage cues from monocular depth
prediction. Then, we train a diffusion model in the 3D-aware latent space,
thereby enabling synthesis of high-quality 3D-consistent image samples,
outperforming recent state-of-the-art GAN-based methods. Importantly, our
3D-aware LDM is trained without any direct supervision from multiview images or
3D geometry and does not require posed images or learned pose or camera
distributions. It directly learns a 3D representation without relying on
canonical camera coordinates. This opens up promising research avenues for
scalable 3D-aware image synthesis and 3D content creation from in-the-wild
image data. See this https URL for videos of our 3D
results.

基于视图空间和潜在扩散模型的 3D 感知图像合成方法，通过压缩潜在表示学习图像的 3D 结构，实现高质量的 3D-consistent 图像合成，无需多视角或 3D 几何的直接监督，不依赖于规范化的相机坐标。