Motion is a salient cue to recognize actions in video. Modern action
recognition models leverage motion information either explicitly by using
optical flow as input or implicitly by means of 3D convolutional filters that
simultaneously capture appearance and motion information. This paper proposes
an alternative approach based on a learnable correlation operator that can be
used to establish frame-toframe matches over convolutional feature maps in the
different layers of the network. The proposed architecture enables the fusion
of this explicit temporal matching information with traditional appearance cues
captured by 2D convolution. Our correlation network compares favorably with
widely-used 3D CNNs for video modeling, and achieves competitive results over
the prominent two-stream network while being much faster to train. We
empirically demonstrate that correlation networks produce strong results on a
variety of video datasets, and outperform the state of the art on four popular
benchmarks for action recognition: Kinetics, Something-Something, Diving48 and
Sports1M.

该文提出了一种基于可学习相关算子的替代方法，用于在视频中识别动作，并成功地应用在行为识别的多个数据集上，表现比流行的二元网络更加出色并且速度更快。