We present a high-fidelity 3D generative adversarial network (GAN) inversion
framework that can synthesize photo-realistic novel views while preserving
specific details of the input image. High-fidelity 3D GAN inversion is
inherently challenging due to the geometry-texture trade-off in 3D inversion,
where overfitting to a single view input image often damages the estimated
geometry during the latent optimization. To solve this challenge, we propose a
novel pipeline that builds on the pseudo-multi-view estimation with visibility
analysis. We keep the original textures for the visible parts and utilize
generative priors for the occluded parts. Extensive experiments show that our
approach achieves advantageous reconstruction and novel view synthesis quality
over state-of-the-art methods, even for images with out-of-distribution
textures. The proposed pipeline also enables image attribute editing with the
inverted latent code and 3D-aware texture modification. Our approach enables
high-fidelity 3D rendering from a single image, which is promising for various
applications of AI-generated 3D content.

提出了一种高保真的三维对抗生成网络，可以在保留输入图像的特定细节的同时合成逼真的新视图。该方法通过伪多视图估计和能见度分析建立，有效解决了三维反演中的几何纹理平衡问题，并成功实现了单张图像的高保真三维重建和新视图合成。并且本方法亦适用于图像属性编辑和 3D 纹理修改，因此有望被广泛应用于人工智能生成的三维内容领域。

基于伪多视角优化的高保真三维 GAN 反演

High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization

In this paper, we present InSeGAN, an unsupervised 3D generative adversarial
network (GAN) for segmenting (nearly) identical instances of rigid objects in
depth images. Using an analysis-by-synthesis approach, we design a novel GAN
architecture to synthesize a multiple-instance depth image with independent
control over each instance. InSeGAN takes in a set of code vectors (e.g.,
random noise vectors), each encoding the 3D pose of an object that is
represented by a learned implicit object template. The generator has two
distinct modules. The first module, the instance feature generator, uses each
encoded pose to transform the implicit template into a feature map
representation of each object instance. The second module, the depth image
renderer, aggregates all of the single-instance feature maps output by the
first module and generates a multiple-instance depth image. A discriminator
distinguishes the generated multiple-instance depth images from the
distribution of true depth images. To use our model for instance segmentation,
we propose an instance pose encoder that learns to take in a generated depth
image and reproduce the pose code vectors for all of the object instances. To
evaluate our approach, we introduce a new synthetic dataset, "Insta-10",
consisting of 100,000 depth images, each with 5 instances of an object from one
of 10 classes. Our experiments on Insta-10, as well as on real-world noisy
depth images, show that InSeGAN achieves state-of-the-art performance, often
outperforming prior methods by large margins.

本文介绍了 InSeGAN：一种用于分割几乎相同的刚性物体的无监督 3D 生成对抗网络，并提出了一种用于实例分割的子模型，基于 Insta-10 这个新数据集的实验证明了其优异性能。

InSeGAN：一种用于深度图像中分割相同实例的生成式方法

InSeGAN: A Generative Approach to Segmenting Identical Instances in  Depth Images

We study the problem of 3D object generation. We propose a novel framework,
namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects
from a probabilistic space by leveraging recent advances in volumetric
convolutional networks and generative adversarial nets. The benefits of our
model are three-fold: first, the use of an adversarial criterion, instead of
traditional heuristic criteria, enables the generator to capture object
structure implicitly and to synthesize high-quality 3D objects; second, the
generator establishes a mapping from a low-dimensional probabilistic space to
the space of 3D objects, so that we can sample objects without a reference
image or CAD models, and explore the 3D object manifold; third, the adversarial
discriminator provides a powerful 3D shape descriptor which, learned without
supervision, has wide applications in 3D object recognition. Experiments
demonstrate that our method generates high-quality 3D objects, and our
unsupervisedly learned features achieve impressive performance on 3D object
recognition, comparable with those of supervised learning methods.

本文提出了一种新颖的框架，即 3D 生成对抗网络（3D-GAN），通过利用体积卷积网络和生成对抗网络的最新进展，从概率空间生成 3D 对象，并具有生成高质量 3D 对象、无先验 CAD 模型或图像依赖性、生成器从低维概率空间到 3D 空间的映射以及训练出的无监督深度学习的 3D 描述符等优点，在 3D 对象生成及识别等方面取得了令人印象深刻的性能。