Natural Language Video Localization (NLVL), grounding phrases from natural
language descriptions to corresponding video segments, is a complex yet
critical task in video understanding. Despite ongoing advancements, many
existing solutions lack the capability to globally capture temporal dynamics of
the video data. In this study, we present a novel approach to NLVL that aims to
address this issue. Our method involves the direct generation of a global 2D
temporal map via a conditional denoising diffusion process, based on the input
video and language query. The main challenges are the inherent sparsity and
discontinuity of a 2D temporal map in devising the diffusion decoder. To
address these challenges, we introduce a multi-scale technique and develop an
innovative diffusion decoder. Our approach effectively encapsulates the
interaction between the query and video data across various time scales.
Experiments on the Charades and DiDeMo datasets underscore the potency of our
design.

自然语言视频定位（NLVL）是视频理解中复杂但关键的任务之一，本研究提出了一种旨在解决全局捕获视频数据时空动态的 NLVL 新方法，通过条件去噪扩散过程直接生成全局 2D 时空图。多尺度技术和创新的扩散解码器克服了 2D 时空图的稀疏性和不连续性，有效地捕捉了查询和视频数据在不同时间尺度下的交互作用。实验证明了我们设计的有效性。

多尺度二维时间地图扩散模型用于自然语言视频定位

Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video  Localization

Zero-shot Natural Language-Video Localization (NLVL) methods have exhibited
promising results in training NLVL models exclusively with raw video data by
dynamically generating video segments and pseudo-query annotations. However,
existing pseudo-queries often lack grounding in the source video, resulting in
unstructured and disjointed content. In this paper, we investigate the
effectiveness of commonsense reasoning in zero-shot NLVL. Specifically, we
present CORONET, a zero-shot NLVL framework that leverages commonsense to
bridge the gap between videos and generated pseudo-queries via a commonsense
enhancement module. CORONET employs Graph Convolution Networks (GCN) to encode
commonsense information extracted from a knowledge graph, conditioned on the
video, and cross-attention mechanisms to enhance the encoded video and
pseudo-query representations prior to localization. Through empirical
evaluations on two benchmark datasets, we demonstrate that CORONET surpasses
both zero-shot and weakly supervised baselines, achieving improvements up to
32.13% across various recall thresholds and up to 6.33% in mIoU. These results
underscore the significance of leveraging commonsense reasoning for zero-shot
NLVL.

本文研究了零样本自然语言视频定位（NLVL）中常识推理的有效性，并提出了一个名为 CORONET 的零样本 NLVL 框架，利用常识知识图谱，通过常识增强模块来弥合视频和生成的伪查询之间的差距。通过对两个基准数据集的实证评估，证明 CORONET 超过了零样本和弱监督的基准，不同召回阈值下的改进幅度高达 32.13％，在 mIoU 方面的改进幅度高达 6.33％，突出了利用常识推理进行零样本 NLVL 的重要性。

零样本自然语言视频定位中的常识

Commonsense for Zero-Shot Natural Language Video Localization

Given a query, the task of Natural Language Video Localization (NLVL) is to
localize a temporal moment in an untrimmed video that semantically matches the
query. In this paper, we adopt a proposal-based solution that generates
proposals (i.e., candidate moments) and then select the best matching proposal.
On top of modeling the cross-modal interaction between candidate moments and
the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient
moment-moment relation modeling. The core idea is to sample a subset of moments
guided by the learnable templates with an adopted DETR (DEtection TRansformer)
framework. To achieve this, we design a multi-scale visual-linguistic encoder,
and an anchor-guided moment decoder paired with a set of learnable templates.
Experimental results on three public datasets demonstrate the superior
performance of MS-DETR.

本文旨在解决 Natural Language Video Localization（NLVL）的问题，通过提出 Moment Sampling DETR 算法，即基于提议的解决方案，生成候选时刻，并选择最佳匹配提议，同时使用可学习的模板进行多尺度视觉和语言编码，对数据集进行实验，提高了 MS-DETR 的准确性。