Neural radiance fields (NeRFs) are promising 3D representations for scenes,
objects, and humans. However, most existing methods require multi-view inputs
and per-scene training, which limits their real-life applications. Moreover,
current methods focus on single-subject cases, leaving scenes of interacting
hands that involve severe inter-hand occlusions and challenging view variations
remain unsolved. To tackle these issues, this paper proposes a generalizable
visibility-aware NeRF (VA-NeRF) framework for interacting hands. Specifically,
given an image of interacting hands as input, our VA-NeRF first obtains a
mesh-based representation of hands and extracts their corresponding geometric
and textural features. Subsequently, a feature fusion module that exploits the
visibility of query points and mesh vertices is introduced to adaptively merge
features of both hands, enabling the recovery of features in unseen areas.
Additionally, our VA-NeRF is optimized together with a novel discriminator
within an adversarial learning paradigm. In contrast to conventional
discriminators that predict a single real/fake label for the synthesized image,
the proposed discriminator generates a pixel-wise visibility map, providing
fine-grained supervision for unseen areas and encouraging the VA-NeRF to
improve the visual quality of synthesized images. Experiments on the
Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms
conventional NeRFs significantly. Project Page:
https://github.com/XuanHuang0/VANeRF.

这篇论文提出了一种可推广的可见性感知交互手部 NeRF（VA-NeRF）框架，通过利用可见性信息和改进的对抗学习，能够更好地还原具有交互的手部场景中不可见区域的特征，从而在 Interhand2.6M 数据集上取得显著的性能提升。

基于三维可见性感知的通用神经辐射场模型用于交互手部

3D Visibility-aware Generalizable Neural Radiance Fields for Interacting  Hands

We propose VisFusion, a visibility-aware online 3D scene reconstruction
approach from posed monocular videos. In particular, we aim to reconstruct the
scene from volumetric features. Unlike previous reconstruction methods which
aggregate features for each voxel from input views without considering its
visibility, we aim to improve the feature fusion by explicitly inferring its
visibility from a similarity matrix, computed from its projected features in
each image pair. Following previous works, our model is a coarse-to-fine
pipeline including a volume sparsification process. Different from their works
which sparsify voxels globally with a fixed occupancy threshold, we perform the
sparsification on a local feature volume along each visual ray to preserve at
least one voxel per ray for more fine details. The sparse local volume is then
fused with a global one for online reconstruction. We further propose to
predict TSDF in a coarse-to-fine manner by learning its residuals across scales
leading to better TSDF predictions. Experimental results on benchmarks show
that our method can achieve superior performance with more scene details. Code
is available at: this https URL

提出了一种名为 VisFusion 的可见性感知在线 3D 场景重建方法，通过对单目视频进行姿态估计来恢复场景的体量特征，并提出了细节保留的局部稀疏化方法和分层预测 TSDF 的方法，实验结果表明我们的方法能够获得卓越的性能。