For video and volumetric data understanding, 3D convolution layers are widely used in deep learning, however, at the cost of increasing computation and training time. Recent works seek to replace the 3D convolution layer with convolution blocks, e.g. structured combinations of 2D and 1D convolution layers. In this paper, we propose a novel convolution block, Parallel Separable 3D Convolution (PmSCn), which applies m parallel streams of n 2D and one 1D convolution layers along different dimensions. We first mathematically justify the need of parallel streams (Pm) to replace a single 3D convolution layer through tensor decomposition. Then we jointly replace consecutive 3D convolution layers, common in modern network architectures, with the multiple 2D convolution layers (Cn). Lastly, we empirically show that PmSCn is applicable to different backbone architectures, such as ResNet, DenseNet, and UNet, for different applications, such as video action recognition, MRI brain segmentation, and electron microscopy segmentation. In all three applications, we replace the 3D convolution layers in state-of-the art models with PmSCn and achieve around 14% improvement in test performance and 40% reduction in model size and on average.

本文通过张量分解，提出了一种新的卷积块——PmSCn，通过 m 个并行流和不同维度的 n 个 2D 卷积层以及一个 1D 卷积层的结构，可以替代现代网络体系结构中常见的连续 3D 卷积层，并在视频识别，MRI大脑分割和电子显微镜分割三个应用中，取得了平均测试性能提高14％和模型尺寸减少40％的良好结果。

用于视频和体积数据理解的并行可分离3D卷积