In this paper, we introduce a new task, spoken video grounding (SVG), which
aims to localize the desired video fragments from spoken language descriptions.
Compared with using text, employing audio requires the model to directly
exploit the useful phonemes and syllables related to the video from raw speech.
Moreover, we randomly add environmental noises to this speech audio, further
increasing the difficulty of this task and better simulating real applications.
To rectify the discriminative phonemes and extract video-related information
from noisy audio, we develop a novel video-guided curriculum learning (VGCL)
during the audio pre-training process, which can make use of the vital visual
perceptions to help understand the spoken language and suppress the external
noise. Considering during inference the model can not obtain ground truth video
segments, we design a curriculum strategy that gradually shifts the input video
from the ground truth to the entire video content during pre-training. Finally,
the model can learn how to extract critical visual information from the entire
video clip to help understand the spoken language. In addition, we collect the
first large-scale spoken video grounding dataset based on ActivityNet, which is
named as ActivityNet Speech dataset. Extensive experiments demonstrate our
proposed video-guided curriculum learning can facilitate the pre-training
process to obtain a mutual audio encoder, significantly promoting the
performance of spoken video grounding tasks. Moreover, we prove that in the
case of noisy sound, our model outperforms the method that grounding video with
ASR transcripts, further demonstrating the effectiveness of our curriculum
strategy.

本文介绍了一种新的任务：口语视频 grounding（SVG），旨在将口语描述中的期望视频片段定位出来。该文通过电视测试数据表明，采用音频需要模型直接从原始语音中利用有用的音素和音节与视频相关信息。为了更好地模拟实际应用，还将环境噪声随机添加到这种语音音频中，用于纠正识别性音素并从噪声音频中提取视频相关信息，研发一种新的视频引导课程学习（VGCL）方法。经过广泛的实验证明，我们提出的视频引导课程学习可能促进预训练过程，从而获得一种相互的音频编码器，并显着促进了口语视频焦点任务的表现。