We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time.

本文介绍了名为NeuralRecon的新型框架，可实时从单目视频中重建3D场景。该系统采用基于学习的TSDF融合模块，通过神经网络直接逐个重建局部表面，从而捕捉局部平滑性先验和全局形状先验，实现高精度，连贯和实时的表面重建。实验结果表明，该系统在准确性和速度方面均优于现有的方法。这是首个能够在实时情况下重建连贯稠密3D几何模型的基于学习的系统。

NeuralRecon：单目视频实时一致性三维重建