This paper presents a novel transformer-based framework for unsupervised
activity segmentation which leverages not only frame-level cues but also
segment-level cues. This is in contrast with previous methods which often rely
on frame-level information only. Our approach begins with a frame-level
prediction module which estimates framewise action classes via a transformer
encoder. The frame-level prediction module is trained in an unsupervised manner
via temporal optimal transport. To exploit segment-level information, we
introduce a segment-level prediction module and a frame-to-segment alignment
module. The former includes a transformer decoder for estimating video
transcripts, while the latter matches frame-level features with segment-level
features, yielding permutation-aware segmentation results. Moreover, inspired
by temporal optimal transport, we develop simple-yet-effective pseudo labels
for unsupervised training of the above modules. Our experiments on four public
datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop
Assembly show that our approach achieves comparable or better performance than
previous methods in unsupervised activity segmentation.

本文提出了一种新颖的基于 Transformer 的框架，用于无监督活动分割，利用帧级别线索和段级别线索。我们的方法通过一个帧级别预测模块开始，该模块通过 Transformer 编码器估计逐帧动作类别。为了利用段级别信息，我们引入了一个段级别预测模块和一个帧到段对齐模块。

基于无监督帧到片段对齐的置换感知动作分割

Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment  Alignment

We present a novel approach for unsupervised activity segmentation which uses
video frame clustering as a pretext task and simultaneously performs
representation learning and online clustering. This is in contrast with prior
works where representation learning and clustering are often performed
sequentially. We leverage temporal information in videos by employing temporal
optimal transport. In particular, we incorporate a temporal regularization term
which preserves the temporal order of the activity into the standard optimal
transport module for computing pseudo-label cluster assignments. The temporal
optimal transport module enables our approach to learn effective
representations for unsupervised activity segmentation. Furthermore, previous
methods require storing learned features for the entire dataset before
clustering them in an offline manner, whereas our approach processes one
mini-batch at a time in an online manner. Extensive evaluations on three public
datasets, i.e. 50-Salads, YouTube Instructions, and Breakfast, and our dataset,
i.e., Desktop Assembly, show that our approach performs on par with or better
than previous methods, despite having significantly less memory constraints.

本文提出了一种新颖的无监督活动分割方法，该方法使用视频帧聚类作为预文本任务，并同时执行表示学习和在线聚类。通过使用时间最优传输来利用视频中的时间信息，同时，在计算伪标签聚类分配的标准最优传输模块中加入保留活动时间顺序的时间正则化项。与之前的方法相比，我们的方法不需要在离线模式下存储整个数据集的特征，而是以在线模式每次处理一个小批量。 extensive evaluations on three public datasets, i.e. 50-Salads, YouTube Instructions, and Breakfast, and our dataset, i.e., Desktop Assembly, show that our approach performs on par with or better than previous methods, despite having significantly less memory constraints.