We address the problem of text-guided video temporal grounding, which aims to
identify the time interval of a certain event based on a natural language
description. Different from most existing methods that only consider RGB images
as visual features, we propose a multi-modal framework to extract complementary
information from videos. Specifically, we adopt RGB images for appearance,
optical flow for motion, and depth maps for image structure. While RGB images
provide abundant visual cues of certain events, the performance may be affected
by background clutters. Therefore, we use optical flow to focus on large motion
and depth maps to infer the scene configuration when the action is related to
objects recognizable with their shapes. To integrate the three modalities more
effectively and enable inter-modal learning, we design a dynamic fusion scheme
with transformers to model the interactions between modalities. Furthermore, we
apply intra-modal self-supervised learning to enhance feature representations
across videos for each modality, which also facilitates multi-modal learning.
We conduct extensive experiments on the Charades-STA and ActivityNet Captions
datasets, and show that the proposed method performs favorably against
state-of-the-art approaches.

本文提出了一种基于多模态框架的文本指导视频时间地基方法，采用 RGB 图像、光流和深度图来提取视频的补充信息并通过 transformers 设计动态融合方案进行交互学习，在 Charades-STA 和 ActivityNet Captions 数据集上实验表明，该方法表现优越。