Temporal convolution has been widely used for video classification. However,
it is performed on spatio-temporal contexts in a limited view, which often
weakens its capacity of learning video representation. To alleviate this
problem, we propose a concise and novel SmallBig network, with the cooperation
of small and big views. For the current time step, the small view branch is
used to learn the core semantics, while the big view branch is used to capture
the contextual semantics. Unlike traditional temporal convolution, the big view
branch can provide the small view branch with the most activated video features
from a broader 3D receptive field. Via aggregating such big-view contexts, the
small view branch can learn more robust and discriminative spatio-temporal
representations for video classification. Furthermore, we propose to share
convolution in the small and big view branch, which improves model compactness
as well as alleviates overfitting. As a result, our SmallBigNet achieves a
comparable model size like 2D CNNs, while boosting accuracy like 3D CNNs. We
conduct extensive experiments on the large-scale video benchmarks, e.g.,
Kinetics400, Something-Something V1 and V2. Our SmallBig network outperforms a
number of recent state-of-the-art approaches, in terms of accuracy and/or
efficiency. The codes and models will be available on
this https URL

提出了一个称为 SmallBig network 的新颖卷积神经网络，它利用小视图和大视图的协作来学习视频表示。相对于传统的时间卷积，大视图可以从更广阔的三维感受野中提供最活跃的视频特征，从而为视频分类学习更加强大和有区别性的时空表示。同时，共享卷积核进一步提高了模型紧凑性并缓解了过拟合。在 Kinetics400、Something-Something V1 和 V2 等大规模视频基准测试中，SmallBig 网络在准确性和效率方面均优于许多最新的先进方法。