Algorithms for the action segmentation task typically use temporal models to
predict what action is occurring at each frame for a minute-long daily
activity. Recent studies have shown the potential of Transformer in modeling
the relations among elements in sequential data. However, there are several
major concerns when directly applying the Transformer to the action
segmentation task, such as the lack of inductive biases with small training
sets, the deficit in processing long input sequence, and the limitation of the
decoder architecture to utilize temporal relations among multiple action
segments to refine the initial predictions. To address these concerns, we
design an efficient Transformer-based model for action segmentation task, named
ASFormer, with three distinctive characteristics: (i) We explicitly bring in
the local connectivity inductive priors because of the high locality of
features. It constrains the hypothesis space within a reliable scope, and is
beneficial for the action segmentation task to learn a proper target function
with small training sets. (ii) We apply a pre-defined hierarchical
representation pattern that efficiently handles long input sequences. (iii) We
carefully design the decoder to refine the initial predictions from the
encoder. Extensive experiments on three public datasets demonstrate that
effectiveness of our methods. Code is available at
https://github.com/ChinaYi/ASFormer.

本文提出了一种名为 ASFormer 的高效 Transformer 模型，通过加入局部连接归纳先验、应用预定的分层表示模式和精心设计的解码器来解决应用 Transformer 模型于动作分割任务时遇到的问题，并在三个公共数据集上进行了广泛实验。