The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.

我们提出了一种名为MGFI的新型多粒度特征交互模块，包括文本帧和词帧，用于视频文本表示对齐，以及一种名为CMFI的文本和音频的跨模态特征交互模块，用于解决视频中帧表达不足的问题。实验结果表明，所提出的方法优于现有的最先进方法。

基于多粒度和多模态特征交互的文本视频检索方法