Video Referring Expression Comprehension (REC) aims to localize a target
object in videos based on the queried natural language. Recent improvements in
video REC have been made using Transformer-based methods with learnable
queries. However, we contend that this naive query design is not ideal given
the open-world nature of video REC brought by text supervision. With numerous
potential semantic categories, relying on only a few slow-updated queries is
insufficient to characterize them. Our solution to this problem is to create
dynamic queries that are conditioned on both the input video and language to
model the diverse objects referred to. Specifically, we place a fixed number of
learnable bounding boxes throughout the frame and use corresponding region
features to provide prior information. Also, we noticed that current query
features overlook the importance of cross-modal alignment. To address this, we
align specific phrases in the sentence with semantically relevant visual areas,
annotating them in existing video datasets (VID-Sentence and VidSTG). By
incorporating these two designs, our proposed model (called ConFormer)
outperforms other models on widely benchmarked datasets. For example, in the
testing split of VID-Sentence dataset, ConFormer achieves 8.75% absolute
improvement on Accu.@0.6 compared to the previous state-of-the-art model.

使用动态查询和跨模态对齐，基于 Transformer 方法的 ConFormer 模型在视频参考表达理解中表现出色，实现了对目标对象的定位。

视频参照表达理解中的基于内容条件查询的 Transformer

Video Referring Expression Comprehension via Transformer with  Content-conditioned Query

We investigate the problem of video Referring Expression Comprehension (REC),
which aims to localize the referent objects described in the sentence to visual
regions in the video frames. Despite the recent progress, existing methods
suffer from two problems: 1) inconsistent localization results across video
frames; 2) confusion between the referent and contextual objects. To this end,
we propose a novel Dual Correspondence Network (dubbed as DCNet) which
explicitly enhances the dense associations in both the inter-frame and
cross-modal manners. Firstly, we aim to build the inter-frame correlations for
all existing instances within the frames. Specifically, we compute the
inter-frame patch-wise cosine similarity to estimate the dense alignment and
then perform the inter-frame contrastive learning to map them close in feature
space. Secondly, we propose to build the fine-grained patch-word alignment to
associate each patch with certain words. Due to the lack of this kind of
detailed annotations, we also predict the patch-word correspondence through the
cosine similarity. Extensive experiments demonstrate that our DCNet achieves
state-of-the-art performance on both video and image REC benchmarks.
Furthermore, we conduct comprehensive ablation studies and thorough analyses to
explore the optimal model designs. Notably, our inter-frame and cross-modal
contrastive losses are plug-and-play functions and are applicable to any video
REC architectures. For example, by building on top of Co-grounding, we boost
the performance by 1.48% absolute improvement on Accu.@0.5 for VID-Sentence
dataset.

研究视频指代表达理解中的两个问题，提出了一种新颖的双重对应网络方法来增强帧间和跨模态的密集关联，以提高视频和图像 REC 基准测试的表现，并进行了全面的剖析研究。