While recent advances in neural radiance field enable realistic digitization
for large-scale scenes, the image-capturing process is still time-consuming and
labor-intensive. Previous works attempt to automate this process using the
Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing
NBV policies heavily rely on hand-crafted criteria, limited action space, or
per-scene optimized representations. These constraints limit their
cross-dataset generalizability. To overcome them, we propose GenNBV, an
end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning
(RL)-based framework and extends typical limited action space to 5D free space.
It empowers our agent drone to scan from any viewpoint, and even interact with
unseen geometries during training. To boost the cross-dataset generalizability,
we also propose a novel multi-source state embedding, including geometric,
semantic, and action representations. We establish a benchmark using the Isaac
Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV
policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12%
coverage ratio on unseen building-scale objects from these datasets,
respectively, outperforming prior solutions.

通过使用深度学习框架与增强学习方法，我们提出了一个新的基于 Next-Best-View 策略的 3D 重建模型，该模型在多个数据集上取得了优异的性能，同时具备跨数据集泛化能力。

GenNBV：通用的主动三维重建最佳观察策略

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

We introduce an active 3D reconstruction method which integrates visual
perception, robot-object interaction, and 3D scanning to recover both the
exterior and interior, i.e., unexposed, geometries of a target 3D object.
Unlike other works in active vision which focus on optimizing camera viewpoints
to better investigate the environment, the primary feature of our
reconstruction is an analysis of the interactability of various parts of the
target object and the ensuing part manipulation by a robot to enable scanning
of occluded regions. As a result, an understanding of part articulations of the
target object is obtained on top of complete geometry acquisition. Our method
operates fully automatically by a Fetch robot with built-in RGBD sensors. It
iterates between interaction analysis and interaction-driven reconstruction,
scanning and reconstructing detected moveable parts one at a time, where both
the articulated part detection and mesh reconstruction are carried out by
neural networks. In the final step, all the remaining, non-articulated parts,
including all the interior structures that had been exposed by prior part
manipulations and subsequently scanned, are reconstructed to complete the
acquisition. We demonstrate the performance of our method via qualitative and
quantitative evaluation, ablation studies, comparisons to alternatives, as well
as experiments in a real environment.

利用感知、机器人对象交互和三维扫描的积极三维重建方法，恢复目标三维物体的外部和内部几何结构，通过分析各部分的互动性和机器人的部分操作来扫描遮挡区域，自动化操作通过内嵌 RGBD 传感器的 Fetch 机器人完成，最后重建所有未暴露的非活动部分和内部结构完成获取。