Video-text retrieval (VTR) is an attractive yet challenging task for
multi-modal understanding, which aims to search for relevant video (text) given
a query (video). Existing methods typically employ completely heterogeneous
visual-textual information to align video and text, whilst lacking the
awareness of homogeneous high-level semantic information residing in both
modalities. To fill this gap, in this work, we propose a novel
visual-linguistic aligning model named HiSE for VTR, which improves the
cross-modal representation by incorporating explicit high-level semantics.
First, we explore the hierarchical property of explicit high-level semantics,
and further decompose it into two levels, i.e. discrete semantics and holistic
semantics. Specifically, for visual branch, we exploit an off-the-shelf
semantic entity predictor to generate discrete high-level semantics. In
parallel, a trained video captioning model is employed to output holistic
high-level semantics. As for the textual modality, we parse the text into three
parts including occurrence, action and entity. In particular, the occurrence
corresponds to the holistic high-level semantics, meanwhile both action and
entity represent the discrete ones. Then, different graph reasoning techniques
are utilized to promote the interaction between holistic and discrete
high-level semantics. Extensive experiments demonstrate that, with the aid of
explicit high-level semantics, our method achieves the superior performance
over state-of-the-art methods on three benchmark datasets, including MSR-VTT,
MSVD and DiDeMo.

本文提出了一种名为 HiSE 的视觉语言对齐模型，通过明确的高层语义信息来提高跨模态表示，结合图形推理技术来促进全局和离散高级语义之间的交互，通过在 MSR-VTT、MSVD 和 DiDeMo 等三个基准数据集上的广泛实验表明，我们的方法在 VTR 上实现了优于现有方法的性能表现。