Creating and editing the shape and color of 3D objects require tremendous
human effort and expertise. Compared to direct manipulation in 3D interfaces,
2D interactions such as sketches and scribbles are usually much more natural
and intuitive for the users. In this paper, we propose a generic multi-modal
generative model that couples the 2D modalities and implicit 3D representations
through shared latent spaces. With the proposed model, versatile 3D generation
and manipulation are enabled by simply propagating the editing from a specific
2D controlling modality through the latent spaces. For example, editing the 3D
shape by drawing a sketch, re-colorizing the 3D surface via painting color
scribbles on the 2D rendering, or generating 3D shapes of a certain category
given one or a few reference images. Unlike prior works, our model does not
require re-training or fine-tuning per editing task and is also conceptually
simple, easy to implement, robust to input domain shifts, and flexible to
diverse reconstruction on partial 2D inputs. We evaluate our framework on two
representative 2D modalities of grayscale line sketches and rendered color
images, and demonstrate that our method enables various shape manipulation and
generation tasks with these 2D modalities.

该论文提出了一种通用的多模态生成模型，通过共享潜在空间将 2D 模态和隐式 3D 表示耦合在一起，实现了通过简单地传播来自特定 2D 控制模态的编辑，从而实现了多样化的 3D 生成和操作。