This paper presents a novel self-supervised temporal video alignment
framework which is useful for several fine-grained human activity understanding
applications. In contrast with the state-of-the-art method of CASA, where
sequences of 3D skeleton coordinates are taken directly as input, our key idea
is to use sequences of 2D skeleton heatmaps as input. Given 2D skeleton
heatmaps, we utilize a video transformer which performs self-attention in the
spatial and temporal domains for extracting effective spatiotemporal and
contextual features. In addition, we introduce simple heatmap augmentation
techniques based on 2D skeletons for self-supervised learning. Despite the lack
of 3D information, our approach achieves not only higher accuracy but also
better robustness against missing and noisy keypoints than CASA. Extensive
evaluations on three public datasets, i.e., Penn Action, IKEA ASM, and H2O,
demonstrate that our approach outperforms previous methods in different
fine-grained human activity understanding tasks, i.e., phase classification,
phase progression, video alignment, and fine-grained frame retrieval.

本论文提出了一种自监督学习的视频对齐框架，利用 2D 骨架热图作为输入，通过自身在空间和时间域内的自注意力机制来提取有效的时空和上下文特征，同时通过基于 2D 骨架的热图增强技术进行自监督学习，取得了比 CASA 更高的准确度和对缺失和噪声关键点更好的稳健性，在不同的细粒度人体活动理解任务中表现优于先前的方法，如阶段分类、阶段进展、视频对齐和细粒度帧检索。