Streaming video recognition reasons about objects and their actions in every
frame of a video. A good streaming recognition model captures both long-term
dynamics and short-term changes of video. Unfortunately, in most existing
methods, the computational complexity grows linearly or quadratically with the
length of the considered dynamics. This issue is particularly pronounced in
transformer-based architectures. To address this issue, we reformulate the
cross-attention in a video transformer through the lens of kernel and apply two
kinds of temporal smoothing kernel: A box kernel or a Laplace kernel. The
resulting streaming attention reuses much of the computation from frame to
frame, and only requires a constant time update each frame. Based on this idea,
we build TeSTra, a Temporal Smoothing Transformer, that takes in arbitrarily
long inputs with constant caching and computing overhead. Specifically, it runs
$6\times$ faster than equivalent sliding-window based transformers with 2,048
frames in a streaming setting. Furthermore, thanks to the increased temporal
span, TeSTra achieves state-of-the-art results on THUMOS'14 and
EPIC-Kitchen-100, two standard online action detection and action anticipation
datasets. A real-time version of TeSTra outperforms all but one prior
approaches on the THUMOS'14 dataset.

通过在视频变压器的跨关注点中引入核的角度并应用两种时间平滑核，重新定义流式视频识别模型，拥有常数时间更新每帧的优势，在 THUMOS'14 和 EPIC-Kitchen-100 数据集上取得了最新的成果。