Recently, large-scale pre-trained language-image models like CLIP have shown
extraordinary capabilities for understanding spatial contents, but naively
transferring such models to video recognition still suffers from unsatisfactory
temporal modeling capabilities. Existing methods insert tunable structures into
or in parallel with the pre-trained model, which either requires
back-propagation through the whole pre-trained model and is thus
resource-demanding, or is limited by the temporal reasoning capability of the
pre-trained structure. In this work, we present DiST, which disentangles the
learning of spatial and temporal aspects of videos. Specifically, DiST uses a
dual-encoder structure, where a pre-trained foundation model acts as the
spatial encoder, and a lightweight network is introduced as the temporal
encoder. An integration branch is inserted between the encoders to fuse
spatio-temporal information. The disentangled spatial and temporal learning in
DiST is highly efficient because it avoids the back-propagation of massive
pre-trained parameters. Meanwhile, we empirically show that disentangled
learning with an extra network for integration benefits both spatial and
temporal understanding. Extensive experiments on five benchmarks show that DiST
delivers better performance than existing state-of-the-art methods by
convincing gaps. When pre-training on the large-scale Kinetics-710, we achieve
89.7% on Kinetics-400 with a frozen ViT-L model, which verifies the scalability
of DiST. Codes and models can be found in
this https URL

DiST 是一种双编码器结构，其中预训练的基础模型充当空间编码器，引入了轻量级网络作为时间编码器，通过插入一个集成分支来融合时空信息，从而实现了视频的空间和时间解耦学习，提高了性能表现。

高效图像到视频迁移学习的空间和时间解耦

Disentangling Spatial and Temporal Learning for Efficient Image-to-Video  Transfer Learning

We propose the Temporal Point Cloud Networks (TPCN), a novel and flexible
framework with joint spatial and temporal learning for trajectory prediction.
Unlike existing approaches that rasterize agents and map information as 2D
images or operate in a graph representation, our approach extends ideas from
point cloud learning with dynamic temporal learning to capture both spatial and
temporal information by splitting trajectory prediction into both spatial and
temporal dimensions. In the spatial dimension, agents can be viewed as an
unordered point set, and thus it is straightforward to apply point cloud
learning techniques to model agents' locations. While the spatial dimension
does not take kinematic and motion information into account, we further propose
dynamic temporal learning to model agents' motion over time. Experiments on the
Argoverse motion forecasting benchmark show that our approach achieves the
state-of-the-art results.

本文介绍了一种新型的灵活的框架 ——Temporal Point Cloud Networks (TPCN)，该框架采用联合的空间和时间学习用于轨迹预测，并将轨迹预测分为空间和时间两个维度。该方法采用点云学习技术模拟智能体的位置，运用动态时间学习模拟智能体的运动。实验结果表明，该方法在 Argoverse 运动预测基准测试中具有最先进的性能。