Spatio-temporal action detection (STAD) is an important fine-grained video
understanding task. Current methods require box and label supervision for all
action classes in advance. However, in real-world applications, it is very
likely to come across new action classes not seen in training because the
action category space is large and hard to enumerate. Also, the cost of data
annotation and model training for new classes is extremely high for traditional
methods, as we need to perform detailed box annotations and re-train the whole
network from scratch. In this paper, we propose a new challenging setting by
performing open-vocabulary STAD to better mimic the situation of action
detection in an open world. Open-vocabulary spatio-temporal action detection
(OV-STAD) requires training a model on a limited set of base classes with box
and label supervision, which is expected to yield good generalization
performance on novel action classes. For OV-STAD, we build two benchmarks based
on the existing STAD datasets and propose a simple but effective method based
on pretrained video-language models (VLM). To better adapt the holistic VLM for
the fine-grained action detection task, we carefully fine-tune it on the
localized video region-text pairs. This customized fine-tuning endows the VLM
with better motion understanding, thus contributing to a more accurate
alignment between video regions and texts. Local region feature and global
video feature fusion before alignment is adopted to further improve the action
detection performance by providing global context. Our method achieves a
promising performance on novel classes.

通过对现有 STAD 数据集建立两个基准，并提出基于预训练视频 - 语言模型的简单而有效的方法，我们在新的类别上取得了令人期待的性能，通过训练一个在有限的基本类别上的模型来具备好的泛化性能。

开放词汇空间时间动作检测

Open-Vocabulary Spatio-Temporal Action Detection

Action understanding has evolved into the era of fine granularity, as most
human behaviors in real life have only minor differences. To detect these
fine-grained actions accurately in a label-efficient way, we tackle the problem
of weakly-supervised fine-grained temporal action detection in videos for the
first time. Without the careful design to capture subtle differences between
fine-grained actions, previous weakly-supervised models for general action
detection cannot perform well in the fine-grained setting. We propose to model
actions as the combinations of reusable atomic actions which are automatically
discovered from data through self-supervised clustering, in order to capture
the commonality and individuality of fine-grained actions. The learnt atomic
actions, represented by visual concepts, are further mapped to fine and coarse
action labels leveraging the semantic label hierarchy. Our approach constructs
a visual representation hierarchy of four levels: clip level, atomic action
level, fine action class level and coarse action class level, with supervision
at each level. Extensive experiments on two large-scale fine-grained video
datasets, FineAction and FineGym, show the benefit of our proposed
weakly-supervised model for fine-grained action detection, and it achieves
state-of-the-art results.

本文提出一种弱监督的方法来检测细粒度视频动作，通过自我监督聚类获取可重复和自动发现的原子动作集合，并结合语义标签层次将原子动作映射到细粒度和粗粒度行动标签，最终构建了四个层次的视频可视化表示层次，在两个大型数据集上实验表明该方法在细粒度动作检测方面取得了最优性能。

基于层级原子动作的细粒度视频弱监督时序动作检测

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Fine-grained action detection is an important task with numerous applications
in robotics and human-computer interaction. Existing methods typically utilize
a two-stage approach including extraction of local spatio-temporal features
followed by temporal modeling to capture long-term dependencies. While most
recent papers have focused on the latter (long-temporal modeling), here, we
focus on producing features capable of modeling fine-grained motion more
efficiently. We propose a novel locally-consistent deformable convolution,
which utilizes the change in receptive fields and enforces a local coherency
constraint to capture motion information effectively. Our model jointly learns
spatio-temporal features (instead of using independent spatial and temporal
streams). The temporal component is learned from the feature space instead of
pixel space, e.g. optical flow. The produced features can be flexibly used in
conjunction with other long-temporal modeling networks, e.g. ST-CNN,
DilatedTCN, and ED-TCN. Overall, our proposed approach robustly outperforms the
original long-temporal models on two fine-grained action datasets: 50 Salads
and GTEA, achieving F1 scores of 80.22% and 75.39% respectively.

本文提出了一种新颖的局部一致可变卷积，利用感受野变化并强制执行局部连贯性约束以有效捕获运动信息，其中时空组件是从特征空间而非像素空间学习的。最终结果表明，所提出的方法在两个动作细粒度数据集（Salads50 和 GTEA）上均显著优于原有的长期时间建模方法，分别取得了 80.22％和 75.39％的 F1 分数。