We tackle the problem of learning an implicit scene representation for 3D
instance segmentation from a sequence of posed RGB images. Towards this, we
introduce 3DIML, a novel framework that efficiently learns a label field that
may be rendered from novel viewpoints to produce view-consistent instance
segmentation masks. 3DIML significantly improves upon training and inference
runtimes of existing implicit scene representation based methods. Opposed to
prior art that optimizes a neural field in a self-supervised manner, requiring
complicated training procedures and loss function design, 3DIML leverages a
two-phase process. The first phase, InstanceMap, takes as input 2D segmentation
masks of the image sequence generated by a frontend instance segmentation
model, and associates corresponding masks across images to 3D labels. These
almost view-consistent pseudolabel masks are then used in the second phase,
InstanceLift, to supervise the training of a neural label field, which
interpolates regions missed by InstanceMap and resolves ambiguities.
Additionally, we introduce InstanceLoc, which enables near realtime
localization of instance masks given a trained label field and an off-the-shelf
image segmentation model by fusing outputs from both. We evaluate 3DIML on
sequences from the Replica and ScanNet datasets and demonstrate 3DIML's
effectiveness under mild assumptions for the image sequences. We achieve a
14-24x speedup over existing implicit scene representation methods with
comparable quality, showcasing its potential to facilitate faster and more
effective 3D scene understanding.

我们提出了一种新的框架 3DIML，用于从一系列姿态 RGB 图像中学习隐式场景表示，以实现 3D 实例分割，该框架可以有效地学习一个标签场，并且可以在新视角中进行渲染，生成视图一致的实例分割掩码。我们通过对 Replica 和 ScanNet 数据集的序列进行评估，展示了 3DIML 在图像序列的轻度假设下的有效性，相比具有相似质量的现有方法，我们取得了 14-24 倍的加速，展示了它在加快和提高 3D 场景理解方面的潜力。

高效三维实例映射和定位

Efficient 3D Instance Mapping and Localization with Neural Fields

Encoding 3D points is one of the primary steps in learning-based implicit
scene representation. Using features that gather information from neighbors
with multi-resolution grids has proven to be the best geometric encoder for
this task. However, prior techniques do not exploit some characteristics of
most objects or scenes, such as surface normals and local smoothness. This
paper is the first to exploit those 3D characteristics in 3D geometric encoders
explicitly. In contrast to prior work on using multiple levels of details,
regular cube grids, and trilinear interpolation, we propose 3D-oriented grids
with a novel cylindrical volumetric interpolation for modeling local planar
invariance. In addition, we explicitly include a local feature aggregation for
feature regularization and smoothing of the cylindrical interpolation features.
We evaluate our approach on ABC, Thingi10k, ShapeNet, and Matterport3D, for
object and scene representation. Compared to the use of regular grids, our
geometric encoder is shown to converge in fewer steps and obtain sharper 3D
surfaces. When compared to the prior techniques, our method gets
state-of-the-art results.

利用邻域多分辨率格点的特征编码法在学习基于隐式场景表示中发挥了重要作用，然而先前的技术没有明确地利用大多数物体或场景的表面法线和局部平滑性特征。本文提出了第一个明确地利用了这些三维特征的三维几何编码器，通过使用新颖的圆柱体积插值来建模局部平面不变性。此外，我们明确地结合了局部特征聚合来对圆柱体插值特征进行正则化和平滑处理。我们在 ABC、Thingi10k、ShapeNet 和 Matterport3D 上评估了我们的方法，用于对象和场景表示。与使用常规格点的方法相比，我们的几何编码器显示出更快的收敛速度，并获得更清晰的三维表面。与先前的技术相比，我们的方法取得了最先进的结果。

三维隐式表示的定向网格编码器

Oriented-grid Encoder for 3D Implicit Representations

Neural Radiance Field (NeRF) has shown impressive performance in novel view
synthesis via implicit scene representation. However, it usually suffers from
poor scalability as requiring densely sampled images for each new scene.
Several studies have attempted to mitigate this problem by integrating
Multi-View Stereo (MVS) technique into NeRF while they still entail a
cumbersome fine-tuning process for new scenes. Notably, the rendering quality
will drop severely without this fine-tuning process and the errors mainly
appear around the high-frequency features. In the light of this observation, we
design WaveNeRF, which integrates wavelet frequency decomposition into MVS and
NeRF to achieve generalizable yet high-quality synthesis without any per-scene
optimization. To preserve high-frequency information when generating 3D feature
volumes, WaveNeRF builds Multi-View Stereo in the Wavelet domain by integrating
the discrete wavelet transform into the classical cascade MVS, which
disentangles high-frequency information explicitly. With that, disentangled
frequency features can be injected into classic NeRF via a novel hybrid neural
renderer to yield faithful high-frequency details, and an intuitive
frequency-guided sampling strategy can be designed to suppress artifacts around
high-frequency regions. Extensive experiments over three widely studied
benchmarks show that WaveNeRF achieves superior generalizable radiance field
modeling when only given three images as input.

WaveNeRF 通过将多视角立体重建技术与神经辐射场（NeRF）相结合，利用小波频率分解在不需要针对每个场景进行优化的情况下实现了高质量的综合合成。