We present a near real-time method for 6-DoF tracking of an unknown object
from a monocular RGBD video sequence, while simultaneously performing neural 3D
reconstruction of the object. Our method works for arbitrary rigid objects,
even when visual texture is largely absent. The object is assumed to be
segmented in the first frame only. No additional information is required, and
no assumption is made about the interaction agent. Key to our method is a
Neural Object Field that is learned concurrently with a pose graph optimization
process in order to robustly accumulate information into a consistent 3D
representation capturing both geometry and appearance. A dynamic pool of posed
memory frames is automatically maintained to facilitate communication between
these threads. Our approach handles challenging sequences with large pose
changes, partial and full occlusion, untextured surfaces, and specular
highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets,
demonstrating that our method significantly outperforms existing approaches.
Project page: this https URL

本研究提出了一种近实时的方法，可以从单目 RGBD 视频序列中对未知物体进行 6 自由度跟踪，并同时进行神经 3D 重建。该方法适用于任意刚体对象，即使视觉纹理基本不存在。该方法的关键在于同时学习神经对象场和姿态图优化过程，以稳健地累积信息以捕捉物体的几何和外观，而不需要其他信息和假设。本方法可以处理具有大的位姿变化，部分遮挡，无纹理表面和高光反射等挑战性序列，而且通过在 HO3D，YCBInEOAT 和 BEHAVE 数据集上的测试，证明本研究方法显著优于现有方法。