We introduce Video Transformer (VidTr) with separable-attention for video
classification. Comparing with commonly used 3D networks, VidTr is able to
aggregate spatio-temporal information via stacked attentions and provide better
performance with higher efficiency. We first introduce the vanilla video
transformer and show that transformer module is able to perform spatio-temporal
modeling from raw pixels, but with heavy memory usage. We then present VidTr
which reduces the memory cost by 3.3$\times$ while keeping the same
performance. To further optimize the model, we propose the standard deviation
based topK pooling for attention ($pool_{topK\_std}$), which reduces the
computation by dropping non-informative features along temporal dimension.
VidTr achieves state-of-the-art performance on five commonly used datasets with
lower computational requirement, showing both the efficiency and effectiveness
of our design. Finally, error analysis and visualization show that VidTr is
especially good at predicting actions that require long-term temporal
reasoning.

通过可分离的注意力机制，利用堆叠的注意力聚合时空信息，高效地实现了视频分类、时空建模以及长期时间推理，并通过优化模型提高了模型效率与性能。