Contrastive language-image pretraining (CLIP) has demonstrated remarkable
success in various image tasks. However, how to extend CLIP with effective
temporal modeling is still an open and crucial problem. Existing factorized or
joint spatial-temporal modeling trades off between the efficiency and
performance. While modeling temporal information within straight through tube
is widely adopted in literature, we find that simple frame alignment already
provides enough essence without temporal attention. To this end, in this paper,
we proposed a novel Implicit Learnable Alignment (ILA) method, which minimizes
the temporal modeling effort while achieving incredibly high performance.
Specifically, for a frame pair, an interactive point is predicted in each
frame, serving as a mutual information rich region. By enhancing the features
around the interactive point, two frames are implicitly aligned. The aligned
features are then pooled into a single token, which is leveraged in the
subsequent spatial self-attention. Our method allows eliminating the costly or
insufficient temporal self-attention in video. Extensive experiments on
benchmarks demonstrate the superiority and generality of our module.
Particularly, the proposed ILA achieves a top-1 accuracy of 88.7% on
Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H. Code is
released at this https URL .

本文提出了一种新颖的隐式学习对齐（ILA）方法，可在视频中实现高效的空间自注意力，避免了昂贵或不充足的时间自注意力。 在 Kinetics-400 上，提出的 ILA 与 Swin-L 和 ViViT-H 相比，仅使用更少的 FLOPs 即可实现 88.7％的 top-1 准确率。

基于可学习对齐的隐式时间建模视频识别

Implicit Temporal Modeling with Learnable Alignment for Video  Recognition

Fall detection holds immense importance in the field of healthcare, where
timely detection allows for instant medical assistance. In this context, we
propose a 3D ConvNet architecture which consists of 3D Inception modules for
fall detection. The proposed architecture is a custom version of Inflated 3D
(I3D) architecture, that takes compressed measurements of video sequence as
spatio-temporal input, obtained from compressive sensing framework, rather than
video sequence as input, as in the case of I3D convolutional neural network.
This is adopted since privacy raises a huge concern for patients being
monitored through these RGB cameras. The proposed framework for fall detection
is flexible enough with respect to a wide variety of measurement matrices. Ten
action classes randomly selected from Kinetics-400 with no fall examples, are
employed to train our 3D ConvNet post compressive sensing with different types
of sensing matrices on the original video clips. Our results show that 3D
ConvNet performance remains unchanged with different sensing matrices. Also,
the performance obtained with Kinetics pre-trained 3D ConvNet on compressively
sensed fall videos from benchmark datasets is better than the state-of-the-art
techniques.

本文提出了一种基于 3D Inception 模块的 3D ConvNet 框架用于跌倒检测，该框架采用压缩感知框架获取视频顺序的压缩测量结果作为空时输入，以解决由 RGB 相机监测患者隐私带来的隐私问题，并且在 Kinetics-400 数据集上，3D ConvNet 表现优于现有技术。