Learning discriminative spatiotemporal representation is the key problem of
video understanding. Recently, Vision Transformers (ViTs) have shown their
power in learning long-term video dependency with self-attention.
Unfortunately, they exhibit limitations in tackling local video redundancy, due
to the blind global comparison among tokens. UniFormer has successfully
alleviated this issue, by unifying convolution and self-attention as a relation
aggregator in the transformer format. However, this model has to require a
tiresome and complicated image-pretraining phrase, before being finetuned on
videos. This blocks its wide usage in practice. On the contrary, open-sourced
ViTs are readily available and well-pretrained with rich image supervision.
Based on these observations, we propose a generic paradigm to build a powerful
family of video networks, by arming the pretrained ViTs with efficient
UniFormer designs. We call this family UniFormerV2, since it inherits the
concise style of the UniFormer block. But it contains brand-new local and
global relation aggregators, which allow for preferable accuracy-computation
balance by seamlessly integrating advantages from both ViTs and UniFormer.
Without any bells and whistles, our UniFormerV2 gets the state-of-the-art
recognition performance on 8 popular video benchmarks, including scene-related
Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something
V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to
achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. Code will be
available at this https URL

本文提出了一种用于构建视频网络家族的通用范例，通过将预训练的 Vision Transformers 与高效的 UniFormer 设计相结合，实现了理想的准确性与计算平衡，并在 8 个常见的视频基准测试中取得了最先进的识别性能。