Videos are more well-organized curated data sources for visual concept
learning than images. Unlike the 2-dimensional images which only involve the
spatial information, the additional temporal dimension bridges and synchronizes
multiple modalities. However, in most video detection benchmarks, these
additional modalities are not fully utilized. For example, EPIC Kitchens is the
largest dataset in first-person (egocentric) vision, yet it still relies on
crowdsourced information to refine the action boundaries to provide
instance-level action annotations.
We explored how to eliminate the expensive annotations in video detection
data which provide refined boundaries. We propose a model to learn from the
narration supervision and utilize multimodal features, including RGB, motion
flow, and ambient sound. Our model learns to attend to the frames related to
the narration label while suppressing the irrelevant frames from being used.
Our experiments show that noisy audio narration suffices to learn a good action
detection model, thus reducing annotation expenses.

提出了一种基于多模态学习和叙述监督的视频检测模型，可以从嘈杂的音频叙述中学习动作检测，从而降低标注的费用。