Recently, video moment retrieval and highlight detection (MR/HD) are being
spotlighted as the demand for video understanding is drastically increased. The
key objective of MR/HD is to localize the moment and estimate clip-wise
accordance level, i.e., saliency score, to the given text query. Although the
recent transformer-based models brought some advances, we found that these
methods do not fully exploit the information of a given query. For example, the
relevance between text query and video contents is sometimes neglected when
predicting the moment and its saliency. To tackle this issue, we introduce
Query-Dependent DETR (QD-DETR), a detection transformer tailored for MR/HD. As
we observe the insignificant role of a given query in transformer
architectures, our encoding module starts with cross-attention layers to
explicitly inject the context of text query into video representation. Then, to
enhance the model's capability of exploiting the query information, we
manipulate the video-query pairs to produce irrelevant pairs. Such negative
(irrelevant) video-query pairs are trained to yield low saliency scores, which
in turn, encourages the model to estimate precise accordance between
query-video pairs. Lastly, we present an input-adaptive saliency predictor
which adaptively defines the criterion of saliency scores for the given
video-query pairs. Our extensive studies verify the importance of building the
query-dependent representation for MR/HD. Specifically, QD-DETR outperforms
state-of-the-art methods on QVHighlights, TVSum, and Charades-STA datasets.
Codes are available at this http URL.

本文提出了一种针对视频检索和重点检测的检测变压器模型 Query-Dependent DETR，通过显式注入文本查询的上下文信息并利用对偶输入对提高模型对查询信息的利用能力来改进现有的变压器模型，其在 QVHighlights、TVSum 和 Charades-STA 数据集上展现出优异的性能。