Video moment localization aims to retrieve the target segment of an untrimmed
video according to the natural language query. Weakly supervised methods gains
attention recently, as the precise temporal location of the target segment is
not always available. However, one of the greatest challenges encountered by
the weakly supervised method is implied in the mismatch between the video and
language induced by the coarse temporal annotations. To refine the
vision-language alignment, recent works contrast the cross-modality
similarities driven by reconstructing masked queries between positive and
negative video proposals. However, the reconstruction may be influenced by the
latent spurious correlation between the unmasked and the masked parts, which
distorts the restoring process and further degrades the efficacy of contrastive
learning since the masked words are not completely reconstructed from the
cross-modality knowledge. In this paper, we discover and mitigate this spurious
correlation through a novel proposed counterfactual cross-modality reasoning
method. Specifically, we first formulate query reconstruction as an aggregated
causal effect of cross-modality and query knowledge. Then by introducing
counterfactual cross-modality knowledge into this aggregation, the spurious
impact of the unmasked part contributing to the reconstruction is explicitly
modeled. Finally, by suppressing the unimodal effect of masked query, we can
rectify the reconstructions of video proposals to perform reasonable
contrastive learning. Extensive experimental evaluations demonstrate the
effectiveness of our proposed method. The code is available at
\href{https://github.com/sLdZ0306/CCR}{this https URL}.

通过提出的反事实跨模态推理方法，我们发现和减轻了视频查询重构中的潜在虚假相关性，以执行合理的对比学习。

弱监督视频时刻定位的反事实跨模态推理

Counterfactual Cross-modality Reasoning for Weakly Supervised Video  Moment Localization

Video moment localization, also known as video moment retrieval, aiming to
search a target segment within a video described by a given natural language
query. Beyond the task of temporal action localization whereby the target
actions are pre-defined, video moment retrieval can query arbitrary complex
activities. In this survey paper, we aim to present a comprehensive review of
existing video moment localization techniques, including supervised, weakly
supervised, and unsupervised ones. We also review the datasets available for
video moment localization and group results of related work. In addition, we
discuss promising future directions for this field, in particular large-scale
datasets and interpretable video moment localization models.

对现有的视频时刻定位技术（包括监督、弱监督和非监督），以及可用于视频时刻定位的数据集进行全面回顾，并讨论了大规模数据集和可解释视频时刻定位模型等未来发展方向。

视频片段定位调查

A Survey on Video Moment Localization

We address the problem of video moment localization with natural language,
i.e. localizing a video segment described by a natural language sentence. While
most prior work focuses on grounding the query as a whole, temporal
dependencies and reasoning between events within the text are not fully
considered. In this paper, we propose a novel Temporal Compositional Modular
Network (TCMN) where a tree attention network first automatically decomposes a
sentence into three descriptions with respect to the main event, context event
and temporal signal. Two modules are then utilized to measure the visual
similarity and location similarity between each segment and the decomposed
descriptions. Moreover, since the main event and context event may rely on
different modalities (RGB or optical flow), we use late fusion to form an
ensemble of four models, where each model is independently trained by one
combination of the visual input. Experiments show that our model outperforms
the state-of-the-art methods on the TEMPO dataset.

本论文提出了 Temporal Compositional Modular Network (TCMN) 模型，该模型结合自然语言描述和视觉信息，通过树形注意力网络自动细分为描述主事件、情境事件和时间信号三部分，再使用两个模块计量视频片段与细分描述间的相似度和位置相似度，通过 late fusion 方法组合 RGB 和光流两种数据进行训练，实验证明此模型在 TEMPO 数据集上表现优于现有方法。