We present a novel framework named NeuralRecon for real-time 3D scene
reconstruction from a monocular video. Unlike previous methods that estimate
single-view depth maps separately on each key-frame and fuse them later, we
propose to directly reconstruct local surfaces represented as sparse TSDF
volumes for each video fragment sequentially by a neural network. A
learning-based TSDF fusion module based on gated recurrent units is used to
guide the network to fuse features from previous fragments. This design allows
the network to capture local smoothness prior and global shape prior of 3D
surfaces when sequentially reconstructing the surfaces, resulting in accurate,
coherent, and real-time surface reconstruction. The experiments on ScanNet and
7-Scenes datasets show that our system outperforms state-of-the-art methods in
terms of both accuracy and speed. To the best of our knowledge, this is the
first learning-based system that is able to reconstruct dense coherent 3D
geometry in real-time.

本文介绍了名为 NeuralRecon 的新型框架，可实时从单目视频中重建 3D 场景。该系统采用基于学习的 TSDF 融合模块，通过神经网络直接逐个重建局部表面，从而捕捉局部平滑性先验和全局形状先验，实现高精度，连贯和实时的表面重建。实验结果表明，该系统在准确性和速度方面均优于现有的方法。这是首个能够在实时情况下重建连贯稠密 3D 几何模型的基于学习的系统。