Spatio-temporal video grounding (or STVG) task aims at locating a
spatio-temporal tube for a specific instance given a text query. Despite
advancements, current methods easily suffer the distractors or heavy object
appearance variations in videos due to insufficient object information from the
text, leading to degradation. Addressing this, we propose a novel framework,
context-guided STVG (CG-STVG), which mines discriminative instance context for
object in videos and applies it as a supplementary guidance for target
localization. The key of CG-STVG lies in two specially designed modules,
including instance context generation (ICG), which focuses on discovering
visual context information (in both appearance and motion) of the instance, and
instance context refinement (ICR), which aims to improve the instance context
from ICG by eliminating irrelevant or even harmful information from the
context. During grounding, ICG, together with ICR, are deployed at each
decoding stage of a Transformer architecture for instance context learning.
Particularly, instance context learned from one decoding stage is fed to the
next stage, and leveraged as a guidance containing rich and discriminative
object feature to enhance the target-awareness in decoding feature, which
conversely benefits generating better new instance context for improving
localization finally. Compared to existing methods, CG-STVG enjoys object
information in text query and guidance from mined instance visual context for
more accurate target localization. In our experiments on three benchmarks,
including HCSTVG-v1/-v2 and VidSTG, CG-STVG sets new state-of-the-arts in
m_tIoU and m_vIoU on all of them, showing its efficacy. The code will be
released at this https URL

提出了一种上下文引导的时空视频定位方法（CG-STVG），通过挖掘视频中的实例上下文信息作为目标定位的辅助指导，并通过消除不相关或有害信息来改进实例上下文，从而提高目标的准确性。在三个基准测试上，包括 HCSTVG-v1/-v2 和 VidSTG，CG-STVG 在 m_tIoU 和 m_vIoU 上都取得了新的最优结果，显示了其有效性。

上下文引导的时空视频定位

Context-Guided Spatio-Temporal Video Grounding

Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of
a queried object according to the given sentence. Currently, most existing
grounding methods are restricted to well-aligned segment-sentence pairs. In
this paper, we explore spatio-temporal video grounding on unaligned data and
multi-form sentences. This challenging task requires to capture critical object
relations to identify the queried target. However, existing approaches cannot
distinguish notable objects and remain in ineffective relation modeling between
unnecessary objects. Thus, we propose a novel object-aware multi-branch
relation network for object-aware relation discovery. Concretely, we first
devise multiple branches to develop object-aware region modeling, where each
branch focuses on a crucial object mentioned in the sentence. We then propose
multi-branch relation reasoning to capture critical object relationships
between the main branch and auxiliary branches. Moreover, we apply a diversity
loss to make each branch only pay attention to its corresponding object and
boost multi-branch learning. The extensive experiments show the effectiveness
of our proposed method.

本文探讨了如何对不对齐的数据和多种形式的句子进行时空视频对齐，通过提出可发现对象之间对象感知关系、对象感知多分支关系网络的方式进行研究，最后得到了一个有效的结果。

基于多分支关系网络的物体感知时空视频定位

Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video  Grounding

In this paper, we consider a novel task, Spatio-Temporal Video Grounding for
Multi-Form Sentences (STVG). Given an untrimmed video and a
declarative/interrogative sentence depicting an object, STVG aims to localize
the spatio-temporal tube of the queried object. STVG has two challenging
settings: (1) We need to localize spatio-temporal object tubes from untrimmed
videos, where the object may only exist in a very small segment of the video;
(2) We deal with multi-form sentences, including the declarative sentences with
explicit objects and interrogative sentences with unknown objects. Existing
methods cannot tackle the STVG task due to the ineffective tube pre-generation
and the lack of object relationship modeling. Thus, we then propose a novel
Spatio-Temporal Graph Reasoning Network (STGRN) for this task. First, we build
a spatio-temporal region graph to capture the region relationships with
temporal object dynamics, which involves the implicit and explicit spatial
subgraphs in each frame and the temporal dynamic subgraph across frames. We
then incorporate textual clues into the graph and develop the multi-step
cross-modal graph reasoning. Next, we introduce a spatio-temporal localizer
with a dynamic selection method to directly retrieve the spatio-temporal tubes
without tube pre-generation. Moreover, we contribute a large-scale video
grounding dataset VidSTG based on video relation dataset VidOR. The extensive
experiments demonstrate the effectiveness of our method.

本文研究了一种新的任务，即用于多形式句子的时空视频基础问题。提出了一种名为 Spatio-Temporal Graph Reasoning Network 的方法来解决这个问题，利用时空区域图来捕捉视频中物体的相关性。实验结果表明了本方法的有效性。