Implicit neural representations have shown powerful capacity in modeling
real-world 3D scenes, offering superior performance in novel view synthesis. In
this paper, we target a more challenging scenario, i.e., joint scene novel view
synthesis and editing based on implicit neural scene representations.
State-of-the-art methods in this direction typically consider building separate
networks for these two tasks (i.e., view synthesis and editing). Thus, the
modeling of interactions and correlations between these two tasks is very
limited, which, however, is critical for learning high-quality scene
representations. To tackle this problem, in this paper, we propose a unified
Neural Radiance Field (NeRF) framework to effectively perform joint scene
decomposition and composition for modeling real-world scenes. The decomposition
aims at learning disentangled 3D representations of different objects and the
background, allowing for scene editing, while scene composition models an
entire scene representation for novel view synthesis. Specifically, with a
two-stage NeRF framework, we learn a coarse stage for predicting a global
radiance field as guidance for point sampling, and in the second fine-grained
stage, we perform scene decomposition by a novel one-hot object radiance field
regularization module and a pseudo supervision via inpainting to handle
ambiguous background regions occluded by objects. The decomposed object-level
radiance fields are further composed by using activations from the
decomposition module. Extensive quantitative and qualitative results show the
effectiveness of our method for scene decomposition and composition,
outperforming state-of-the-art methods for both novel-view synthesis and
editing tasks.

本论文提出了一个统一的神经辐射场（NeRF）框架，以有效地进行场景分解和合成，用于建模现实世界场景，通过学习不同对象的解耦 3D 表示进行场景编辑，同时为新颖视角合成建模整个场景表示。

可编辑的小说视角合成的统一分解和组合 NeRF 学习

Learning Unified Decompositional and Compositional NeRF for Editable  Novel View Synthesis

We present a method for composing photorealistic scenes from captured images
of objects. Our work builds upon neural radiance fields (NeRFs), which
implicitly model the volumetric density and directionally-emitted radiance of a
scene. While NeRFs synthesize realistic pictures, they only model static scenes
and are closely tied to specific imaging conditions. This property makes NeRFs
hard to generalize to new scenarios, including new lighting or new arrangements
of objects. Instead of learning a scene radiance field as a NeRF does, we
propose to learn object-centric neural scattering functions (OSFs), a
representation that models per-object light transport implicitly using a
lighting- and view-dependent neural network. This enables rendering scenes even
when objects or lights move, without retraining. Combined with a volumetric
path tracing procedure, our framework is capable of rendering both intra- and
inter-object light transport effects including occlusions, specularities,
shadows, and indirect illumination. We evaluate our approach on scene
composition and show that it generalizes to novel illumination conditions,
producing photorealistic, physically accurate renderings of multi-object
scenes.

提出一种基于神经辐射场和基于对象的神经散射函数的方法，能够在不重新训练的情况下，对静态及动态场景进行光线追踪模拟，拥有在新光照和新物品排列情况下泛化能力的物理准确的多物体场景组成渲染技术。

基于对象的神经场景渲染

Object-Centric Neural Scene Rendering

Structured representations such as scene graphs serve as an efficient and
compact representation that can be used for downstream rendering or retrieval
tasks. However, existing efforts to generate realistic images from scene graphs
perform poorly on scene composition for cluttered or complex scenes. We propose
two contributions to improve the scene composition. First, we enhance the scene
graph representation with heuristic-based relations, which add minimal storage
overhead. Second, we use extreme points representation to supervise the
learning of the scene composition network. These methods achieve significantly
higher performance over existing work (69.0% vs 51.2% in relation score
metric). We additionally demonstrate how scene graphs can be used to retrieve
pose-constrained image patches that are semantically similar to the source
query. Improving structured scene graph representations for rendering or
retrieval is an important step towards realistic image generation.

提出了两种改进场景结构表示法的方法，通过启发式关系和极端点表示法，显著提高相关分数测量方法的性能，并演示了如何使用场景图检索与源查询语义相似的姿态约束图像块，这是实现逼真图像生成的重要一步。