Fusion of 2D images and 3D point clouds is important because information from
dense images can enhance sparse point clouds. However, fusion is challenging
because 2D and 3D data live in different spaces. In this work, we propose
MVPNet (Multi-View PointNet), where we aggregate 2D multi-view image features
into 3D point clouds, and then use a point based network to fuse the features
in 3D canonical space to predict 3D semantic labels. To this end, we introduce
view selection along with a 2D-3D feature aggregation module. Extensive
experiments show the benefit of leveraging features from dense images and
reveal superior robustness to varying point cloud density compared to 3D-only
methods. On the ScanNetV2 benchmark, our MVPNet significantly outperforms prior
point cloud based approaches on the task of 3D Semantic Segmentation. It is
much faster to train than the large networks of the sparse voxel approach. We
provide solid ablation studies to ease the future design of 2D-3D fusion
methods and their extension to other tasks, as we showcase for 3D instance
segmentation.

本文介绍了 MVPNet 模型，该模型使用视图选择和 2D-3D 特征聚合模块将 2D 多视图图像特征汇集到 3D 点云中，并使用基于点的网络将 3D 规范空间中的特征融合以预测 3D 语义标签。该方法在 3D 语义分割任务上，优于以前的点云方法，并且训练速度比稀疏体素方法的大型网络快得多。