In this paper, we efficiently transfer the surpassing representation power of
the vision foundation models, such as ViT and Swin, for video understanding
with only a few trainable parameters. Previous adaptation methods have
simultaneously considered spatial and temporal modeling with a unified
learnable module but still suffered from fully leveraging the representative
capabilities of image transformers. We argue that the popular dual-path
(two-stream) architecture in video models can mitigate this problem. We propose
a novel DualPath adaptation separated into spatial and temporal adaptation
paths, where a lightweight bottleneck adapter is employed in each transformer
block. Especially for temporal dynamic modeling, we incorporate consecutive
frames into a grid-like frameset to precisely imitate vision transformers'
capability that extrapolates relationships between tokens. In addition, we
extensively investigate the multiple baselines from a unified perspective in
video understanding and compare them with DualPath. Experimental results on
four action recognition benchmarks prove that pretrained image transformers
with DualPath can be effectively generalized beyond the data domain.

本文提出了一种新的 DualPath 适应性算法，将图像 transformers 的表示能力有效地转移至视频理解中，包括空间和时间建模，极大地拓展了预训练表示的泛化，同时在四个动作识别基准测试上进行了广泛实验比较。