Developing end-to-end models for long-video action understanding tasks
presents significant computational and memory challenges. Existing works
generally build models on long-video features extracted by off-the-shelf action
recognition models, which are trained on short-video datasets in different
domains, making the extracted features suffer domain discrepancy. To avoid
this, action recognition models can be end-to-end trained on clips, which are
trimmed from long videos and labeled using action interval annotations. Such
fully supervised annotations are expensive to collect. Thus, a weakly
supervised method is needed for long-video action understanding at scale. Under
the weak supervision setting, action labels are provided for the whole video
without precise start and end times of the action clip. To this end, we propose
an AdaFocus framework. AdaFocus estimates the spike-actionness and temporal
positions of actions, enabling it to adaptively focus on action clips that
facilitate better training without the need for precise annotations.
Experiments on three long-video datasets show its effectiveness. Remarkably, on
two of datasets, models trained with AdaFocus under weak supervision outperform
those trained under full supervision. Furthermore, we form a weakly supervised
feature extraction pipeline with our AdaFocus, which enables significant
improvements on three long-video action understanding tasks.

开发用于长视频动作理解任务的端到端模型存在显著的计算和内存挑战。为了避免精确注释的需要，我们提出了一种弱监督方法 AdaFocus，它通过估计动作的高峰度和时间位置，从而能够自适应地聚焦于有助于更好训练的动作片段。在三个长视频数据集上的实验证明了其有效性，并且 AdaFocus 与弱监督特征提取流水线结合，显著提高了三个长视频动作理解任务。