Adapting image models to video domain is becoming an efficient paradigm for
solving video recognition tasks. Due to the huge number of parameters and
effective transferability of image models, performing full fine-tuning is less
efficient and even unnecessary. Thus, recent research is shifting its focus
towards parameter-efficient image-to-video adaptation. However, these
adaptation strategies inevitably introduce extra computational cost to deal
with the domain gap and temporal modeling in videos. In this paper, our goal is
to present a zero-cost adaptation paradigm (ZeroI2V) to transfer the image
transformers to video recognition tasks (i.e., introduce zero extra cost to the
adapted models during inference). To achieve this goal, we present two core
designs. First, to capture the dynamics in videos and reduce the difficulty of
achieving image-to-video adaptation, we exploit the flexibility of
self-attention and introduce the spatial-temporal dual-headed attention (STDHA)
that efficiently endow the image transformers with temporal modeling capability
at zero extra parameters and computation. Second, to handle the domain gap
between images and videos, we propose a linear adaption strategy which utilizes
lightweight densely placed linear adapters to fully transfer the frozen image
models to video recognition. Due to its customized linear design, all newly
added adapters could be easily merged with the original modules through
structural reparameterization after training, thus achieving zero extra cost
during inference. Extensive experiments on four widely-used video recognition
benchmarks show that our ZeroI2V can match or even outperform previous
state-of-the-art methods while enjoying superior parameter and inference
efficiency.

提出了一种零成本适应范式（ZeroI2V）来将图像变换器应用于视频识别任务，实现图像到视频的快速转换和零额外计算成本。通过引入空间 - 时间双头注意力（STDHA）和轻量级线性适配器的线性适应策略，实现了对视频动态性和图像与视频之间的领域差异的处理，达到与先前最先进的方法相当甚至超过的参数和推理效率。

ZeroI2V：从图像到视频的零成本预训练 Transformer 适应

ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to  Video

In this paper, we efficiently transfer the surpassing representation power of
the vision foundation models, such as ViT and Swin, for video understanding
with only a few trainable parameters. Previous adaptation methods have
simultaneously considered spatial and temporal modeling with a unified
learnable module but still suffered from fully leveraging the representative
capabilities of image transformers. We argue that the popular dual-path
(two-stream) architecture in video models can mitigate this problem. We propose
a novel DualPath adaptation separated into spatial and temporal adaptation
paths, where a lightweight bottleneck adapter is employed in each transformer
block. Especially for temporal dynamic modeling, we incorporate consecutive
frames into a grid-like frameset to precisely imitate vision transformers'
capability that extrapolates relationships between tokens. In addition, we
extensively investigate the multiple baselines from a unified perspective in
video understanding and compare them with DualPath. Experimental results on
four action recognition benchmarks prove that pretrained image transformers
with DualPath can be effectively generalized beyond the data domain.

本文提出了一种新的 DualPath 适应性算法，将图像 transformers 的表示能力有效地转移至视频理解中，包括空间和时间建模，极大地拓展了预训练表示的泛化，同时在四个动作识别基准测试上进行了广泛实验比较。