We present Point-Voxel CNN (PVCNN) for efficient, fast 3D deep learning.
Previous work processes 3D data using either voxel-based or point-based NN
models. However, both approaches are computationally inefficient. The
computation cost and memory footprints of the voxel-based models grow cubically
with the input resolution, making it memory-prohibitive to scale up the
resolution. As for point-based networks, up to 80% of the time is wasted on
structuring the sparse data which have rather poor memory locality, not on the
actual feature extraction. In this paper, we propose PVCNN that represents the
3D input data in points to reduce the memory consumption, while performing the
convolutions in voxels to reduce the irregular, sparse data access and improve
the locality. Our PVCNN model is both memory and computation efficient.
Evaluated on semantic and part segmentation datasets, it achieves much higher
accuracy than the voxel-based baseline with 10x GPU memory reduction; it also
outperforms the state-of-the-art point-based models with 7x measured speedup on
average. Remarkably, the narrower version of PVCNN achieves 2x speedup over
PointNet (an extremely efficient model) on part and scene segmentation
benchmarks with much higher accuracy. We validate the general effectiveness of
PVCNN on 3D object detection: by replacing the primitives in Frustrum PointNet
with PVConv, it outperforms Frustrum PointNet++ by 2.4% mAP on average with
1.5x measured speedup and GPU memory reduction.

本文提出了一种称为 Point-Voxel CNN（PVCNN）的算法，其使用点的形式来表示三维输入数据以减少内存消耗，并通过使用体素来执行卷积以提高效率。该算法在语义分割数据集上实现了比基线模型更高的准确性，同时还在速度上超过了最先进的基于点的模型。