We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the per-frame alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1% of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.

我们提出了一种弱监督的框架，用于视频中的动作标注，其中只在训练时间需要发生动作的顺序。通过引入扩展的连接组态分类（ECTC）框架来解决每帧对齐的问题，从而有效地评估所有可能的对齐。我们进一步将我们的框架扩展到在视频中稀疏地注释了一些帧的半监督情况下。使用不到1％的标记帧数的情况下，我们的方法能够胜过现有的半监督方法，并实现与完全监督方法相当的性能。

弱监督动作标注的连接时序建模