We present an end-to-end 3D reconstruction method for a scene by directly
regressing a truncated signed distance function (TSDF) from a set of posed RGB
images. Traditional approaches to 3D reconstruction rely on an intermediate
representation of depth maps prior to estimating a full 3D model of a scene. We
hypothesize that a direct regression to 3D is more effective. A 2D CNN extracts
features from each image independently which are then back-projected and
accumulated into a voxel volume using the camera intrinsics and extrinsics.
After accumulation, a 3D CNN refines the accumulated features and predicts the
TSDF values. Additionally, semantic segmentation of the 3D model is obtained
without significant computation. This approach is evaluated on the Scannet
dataset where we significantly outperform state-of-the-art baselines (deep
multiview stereo followed by traditional TSDF fusion) both quantitatively and
qualitatively. We compare our 3D semantic segmentation to prior methods that
use a depth sensor since no previous work attempts the problem with only RGB
input.

提出了一种端到端的三维重建方法，通过直接回归截断带符号距离函数（TSDF）从一组姿势 RGB 图像中实现。使用 2D CNN 独立地提取每个图像的特征，然后通过相机内参和外参进行反投影和累积到体素体积中。此外，通过该方法获取到了三维模型的语义分割。该方法在 Scannet 数据集上得到了评估，既在量化指标上，也在视觉效果上都超过了国际领先的基准结果。