Prior works on action representation learning mainly focus on designing
various architectures to extract the global representations for short video
clips. In contrast, many practical applications such as video alignment have
strong demand for learning dense representations for long videos. In this
paper, we introduce a novel contrastive action representation learning (CARL)
framework to learn frame-wise action representations, especially for long
videos, in a self-supervised manner. Concretely, we introduce a simple yet
efficient video encoder that considers spatio-temporal context to extract
frame-wise representations. Inspired by the recent progress of self-supervised
learning, we present a novel sequence contrastive loss (SCL) applied on two
correlated views obtained through a series of spatio-temporal data
augmentations. SCL optimizes the embedding space by minimizing the
KL-divergence between the sequence similarity of two augmented views and a
prior Gaussian distribution of timestamp distance. Experiments on FineGym,
PennAction and Pouring datasets show that our method outperforms previous
state-of-the-art by a large margin for downstream fine-grained action
classification. Surprisingly, although without training on paired videos, our
approach also shows outstanding performance on video alignment and fine-grained
frame retrieval tasks. Code and models are available at
this https URL

本文提出了一种新颖的对比行动表示学习（CARL）框架，用于以自我监督的方式学习帧级行动表示，特别是针对长视频。该框架包括一个简单而高效的视频编码器，以及应用于一系列时空数据增强的新颖序列对比损失（SCL）。我们通过 FineGym，PennAction 和 Pouring 数据集的实验证明，该方法在下游的细粒度行动分类任务上表现出明显的优越性。 令人惊讶的是，即使没有对配对视频进行训练，我们的方法在视频对齐和细粒度帧检索任务上也表现出了出色的性能。

通过序列对比学习获取长视频的逐帧动作表示

Frame-wise Action Representations for Long Videos via Sequence  Contrastive Learning

Temporal action segmentation classifies the action of each frame in (long)
video sequences. Due to the high cost of frame-wise labeling, we propose the
first semi-supervised method for temporal action segmentation. Our method
hinges on unsupervised representation learning, which, for temporal action
segmentation, poses unique challenges. Actions in untrimmed videos vary in
length and have unknown labels and start/end times. Ordering of actions across
videos may also vary. We propose a novel way to learn frame-wise
representations from temporal convolutional networks (TCNs) by clustering input
features with added time-proximity condition and multi-resolution similarity.
By merging representation learning with conventional supervised learning, we
develop an "Iterative-Contrast-Classify (ICC)" semi-supervised learning scheme.
With more labelled data, ICC progressively improves in performance; ICC
semi-supervised learning, with 40% labelled videos, performs similar to
fully-supervised counterparts. Our ICC improves MoF by {+1.8, +5.6, +2.5}% on
Breakfast, 50Salads and GTEA respectively for 100% labelled videos.

本论文提出一种基于卷积网络和聚类的半监督学习方法，提高了时间动作分割的性能，其中包括无监督表示学习、带时间接近度限制和多分辨率相似性的特征聚类、以及迭代对比分类等方法。