While current multi-frame restoration methods combine information from
multiple input images using 2D alignment techniques, recent advances in novel
view synthesis are paving the way for a new paradigm relying on volumetric
scene representations. In this work, we introduce the first 3D-based
multi-frame denoising method that significantly outperforms its 2D-based
counterparts with lower computational requirements. Our method extends the
multiplane image (MPI) framework for novel view synthesis by introducing a
learnable encoder-renderer pair manipulating multiplane representations in
feature space. The encoder fuses information across views and operates in a
depth-wise manner while the renderer fuses information across depths and
operates in a view-wise manner. The two modules are trained end-to-end and
learn to separate depths in an unsupervised way, giving rise to Multiplane
Feature (MPF) representations. Experiments on the Spaces and Real
Forward-Facing datasets as well as on raw burst data validate our approach for
view synthesis, multi-frame denoising, and view synthesis under noisy
conditions.

本文介绍了第一种基于三维场景表示的多帧去噪方法，扩展了多平面图像 (MPI) 框架用于新视角合成，最终实现了视角合成、多帧去噪和在嘈杂条件下的视角合成。

基于多平面特征表示的高效视角合成和基于 3D 的多帧降噪

Efficient View Synthesis and 3D-based Multi-Frame Denoising with  Multiplane Feature Representations

Recent work has demonstrated that volumetric scene representations combined
with differentiable volume rendering can enable photo-realistic rendering for
challenging scenes that mesh reconstruction fails on. However, these methods
entangle geometry and appearance in a "black-box" volume that cannot be edited.
Instead, we present an approach that explicitly disentangles
geometry--represented as a continuous 3D volume--from appearance--represented
as a continuous 2D texture map. We achieve this by introducing a 3D-to-2D
texture mapping (or surface parameterization) network into volumetric
representations. We constrain this texture mapping network using an additional
2D-to-3D inverse mapping network and a novel cycle consistency loss to make 3D
surface points map to 2D texture points that map back to the original 3D
points. We demonstrate that this representation can be reconstructed using only
multi-view image supervision and generates high-quality rendering results. More
importantly, by separating geometry and texture, we allow users to edit
appearance by simply editing 2D texture maps.

该研究介绍了一种通过将几何图形和外观区分开来的方法，使用体积场景表示和可微分体积渲染，利用表面参数化网络以及循环一致性损失等技术，生成高质量的渲染结果，且可直接通过编辑 2D 纹理贴图来修改外观。

NeuTex: 用于体积神经渲染的神经纹理映射

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

We propose a novel method for instance label segmentation of dense 3D voxel
grids. We target volumetric scene representations, which have been acquired
with depth sensors or multi-view stereo methods and which have been processed
with semantic 3D reconstruction or scene completion methods. The main task is
to learn shape information about individual object instances in order to
accurately separate them, including connected and incompletely scanned objects.
We solve the 3D instance-labeling problem with a multi-task learning strategy.
The first goal is to learn an abstract feature embedding, which groups voxels
with the same instance label close to each other while separating clusters with
different instance labels from each other. The second goal is to learn instance
information by densely estimating directional information of the instance's
center of mass for each voxel. This is particularly useful to find instance
boundaries in the clustering post-processing step, as well as, for scoring the
segmentation quality for the first goal. Both synthetic and real-world
experiments demonstrate the viability and merits of our approach. In fact, it
achieves state-of-the-art performance on the ScanNet 3D instance segmentation
benchmark.

提出了一种多任务学习策略来解决三维实体标记问题，该方法可以 accurately separate 连通的和扫描不完整的对象，实验表明该方法在 ScanNet 数据集上实现了最先进的性能。