Although the performance of Temporal Action Segmentation (TAS) has improved
in recent years, achieving promising results often comes with a high
computational cost due to dense inputs, complex model structures, and
resource-intensive post-processing requirements. To improve the efficiency
while keeping the performance, we present a novel perspective centered on
per-segment classification. By harnessing the capabilities of Transformers, we
tokenize each video segment as an instance token, endowed with intrinsic
instance segmentation. To realize efficient action segmentation, we introduce
BaFormer, a boundary-aware Transformer network. It employs instance queries for
instance segmentation and a global query for class-agnostic boundary
prediction, yielding continuous segment proposals. During inference, BaFormer
employs a simple yet effective voting strategy to classify boundary-wise
segments based on instance segmentation. Remarkably, as a single-stage
approach, BaFormer significantly reduces the computational costs, utilizing
only 6% of the running time compared to state-of-the-art method DiffAct, while
producing better or comparable accuracy over several popular benchmarks. The
code for this project is publicly available at
this https URL

通过使用 BaFormer 方法，本研究提出了一种基于时间段分类的新视角，以实现高效的动作分割，过程中运用了 Transformer 的优势，将每个视频段作为实例标记，并赋予了内在的实例分割能力，通过使用全局查询进行类别无关的边界预测，得到连续的时间段建议。