Content creation, central to applications such as virtual reality, can be a
tedious and time-consuming. Recent image synthesis methods simplify this task
by offering tools to generate new views from as little as a single input image,
or by converting a semantic map into a photorealistic image. We propose to push
the envelope further, and introduce Generative View Synthesis (GVS), which can
synthesize multiple photorealistic views of a scene given a single semantic
map. We show that the sequential application of existing techniques, e.g.,
semantics-to-image translation followed by monocular view synthesis, fail at
capturing the scene's structure. In contrast, we solve the semantics-to-image
translation in concert with the estimation of the 3D layout of the scene, thus
producing geometrically consistent novel views that preserve semantic
structures. We first lift the input 2D semantic map onto a 3D layered
representation of the scene in feature space, thereby preserving the semantic
labels of 3D geometric structures. We then project the layered features onto
the target views to generate the final novel-view images. We verify the
strengths of our method and compare it with several advanced baselines on three
different datasets. Our approach also allows for style manipulation and image
editing operations, such as the addition or removal of objects, with simple
manipulations of the input style images and semantic maps respectively. Visit
the project page at this https URL

本文提出一种叫做 GVS 的方法，可基于输入的语义地图，综合出多个真实感的场景视角，还可进行风格操作和图像编辑操作，如通过简单处理输入的风格图像和语义映射来添加或删除对象等。

生成视角合成：从单视角语义到新视角图像

Generative View Synthesis: From Single-view Semantics to Novel-view  Images

We present a new point-based approach for modeling the appearance of real
scenes. The approach uses a raw point cloud as the geometric representation of
a scene, and augments each point with a learnable neural descriptor that
encodes local geometry and appearance. A deep rendering network is learned in
parallel with the descriptors, so that new views of the scene can be obtained
by passing the rasterizations of a point cloud from new viewpoints through this
network. The input rasterizations use the learned descriptors as point
pseudo-colors. We show that the proposed approach can be used for modeling
complex scenes and obtaining their photorealistic views, while avoiding
explicit surface estimation and meshing. In particular, compelling results are
obtained for scene scanned using hand-held commodity RGB-D sensors as well as
standard RGB cameras even in the presence of objects that are challenging for
standard mesh-based modeling.

本研究提出了一种基于点的方法，使用可学习的神经描述符来编码局部几何和外观信息，并学习深度渲染网络生成具有照片般逼真感觉的场景视图，同时避免显式表面估计和网格化，并适用于各种复杂场景，包括使用 RGB-D 传感器与标准 RGB 相机进行扫描的场景。