Video temporal dynamics is conventionally modeled with 3D spatial-temporal
kernel or its factorized version comprised of 2D spatial kernel and 1D temporal
kernel. The modeling power, nevertheless, is limited by the fixed window size
and static weights of a kernel along the temporal dimension. The pre-determined
kernel size severely limits the temporal receptive fields and the fixed weights
treat each spatial location across frames equally, resulting in sub-optimal
solution for long-range temporal modeling in natural scenes. In this paper, we
present a new recipe of temporal feature learning, namely Dynamic Temporal
Filter (DTF), that novelly performs spatial-aware temporal modeling in
frequency domain with large temporal receptive field. Specifically, DTF
dynamically learns a specialized frequency filter for every spatial location to
model its long-range temporal dynamics. Meanwhile, the temporal feature of each
spatial location is also transformed into frequency feature spectrum via 1D
Fast Fourier Transform (FFT). The spectrum is modulated by the learnt frequency
filter, and then transformed back to temporal domain with inverse FFT. In
addition, to facilitate the learning of frequency filter in DTF, we perform
frame-wise aggregation to enhance the primary temporal feature with its
temporal neighbors by inter-frame correlation. It is feasible to plug DTF block
into ConvNets and Transformer, yielding DTF-Net and DTF-Transformer. Extensive
experiments conducted on three datasets demonstrate the superiority of our
proposals. More remarkably, DTF-Transformer achieves an accuracy of 83.5% on
Kinetics-400 dataset. Source code is available at
https://github.com/FuchenUSTC/DTF.

该研究提出了一种名为 Dynamic Temporal Filter (DTF) 的方法，能够在频率域进行空间感知的时间建模，并具有较大的时间感受野，可将 DTF 模块嵌入 ConvNets 和 Transformer 中，实验结果表明其优越性。