We propose a weakly-supervised framework for action labeling in video, where
only the order of occurring actions is required during training time. The key
challenge is that the per-frame alignments between the input (video) and label
(action) sequences are unknown during training. We address this by introducing
the Extended Connectionist Temporal Classification (ECTC) framework to
efficiently evaluate all possible alignments via dynamic programming and
explicitly enforce their consistency with frame-to-frame visual similarities.
This protects the model from distractions of visually inconsistent or
degenerated alignments without the need of temporal supervision. We further
extend our framework to the semi-supervised case when a few frames are sparsely
annotated in a video. With less than 1% of labeled frames per video, our method
is able to outperform existing semi-supervised approaches and achieve
comparable performance to that of fully supervised approaches.

我们提出了一种弱监督的框架，用于视频中的动作标注，其中只在训练时间需要发生动作的顺序。通过引入扩展的连接组态分类（ECTC）框架来解决每帧对齐的问题，从而有效地评估所有可能的对齐。我们进一步将我们的框架扩展到在视频中稀疏地注释了一些帧的半监督情况下。使用不到 1％的标记帧数的情况下，我们的方法能够胜过现有的半监督方法，并实现与完全监督方法相当的性能。