This paper addresses unsupervised action segmentation. Prior work captures
the frame-level temporal structure of videos by a feature embedding that
encodes time locations of frames in the video. We advance prior work with a new
self-supervised learning (SSL) of a feature embedding that accounts for both
frame- and action-level structure of videos. Our SSL trains an RNN to recognize
positive and negative action sequences, and the RNN's hidden layer is taken as
our new action-level feature embedding. The positive and negative sequences
consist of action segments sampled from videos, where in the former the sampled
action segments respect their time ordering in the video, and in the latter
they are shuffled. As supervision of actions is not available and our SSL
requires access to action segments, we specify an HMM that explicitly models
action lengths, and infer a MAP action segmentation with the Viterbi algorithm.
The resulting action segmentation is used as pseudo-ground truth for estimating
our action-level feature embedding and updating the HMM. We alternate the above
steps within the Generalized EM framework, which ensures convergence. Our
evaluation on the Breakfast, YouTube Instructions, and 50Salads datasets gives
superior results to those of the state of the art.

本文介绍了一种使用自监督学习和 RNN/HMM 算法实现无监督动作分割的方法，该方法优于现有技术在多个数据集上的表现。