Video moment retrieval (MR) and highlight detection (HD) based on natural
language queries are two highly related tasks, which aim to obtain relevant
moments within videos and highlight scores of each video clip. Recently,
several methods have been devoted to building DETR-based networks to solve both
MR and HD jointly. These methods simply add two separate task heads after
multi-modal feature extraction and feature interaction, achieving good
performance. Nevertheless, these approaches underutilize the reciprocal
relationship between two tasks. In this paper, we propose a task-reciprocal
transformer based on DETR (TR-DETR) that focuses on exploring the inherent
reciprocity between MR and HD. Specifically, a local-global multi-modal
alignment module is first built to align features from diverse modalities into
a shared latent space. Subsequently, a visual feature refinement is designed to
eliminate query-irrelevant information from visual features for modal
interaction. Finally, a task cooperation module is constructed to refine the
retrieval pipeline and the highlight score prediction process by utilizing the
reciprocity between MR and HD. Comprehensive experiments on QVHighlights,
Charades-STA and TVSum datasets demonstrate that TR-DETR outperforms existing
state-of-the-art methods. Codes are available at
https://github.com/mingyao1120/TR-DETR.

基于自然语言查询的视频时刻检索（MR）和精彩片段检测（HD）是两个高度相关的任务，目标是获取视频中的相关时刻和每个视频片段的精彩得分。本文提出了一种基于 DETR 的任务互补变换器（TR-DETR），专注于探索 MR 和 HD 之间的内在互补性。实验表明，TR-DETR 优于现有的最先进方法。

TR-DETR: 具有任务互逆性的 Transformer 模型，用于联合出现时刻检索和摘要提取

TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and  Highlight Detection

Cascade is a classic yet powerful architecture that has boosted performance
on various tasks. However, how to introduce cascade to instance segmentation
remains an open question. A simple combination of Cascade R-CNN and Mask R-CNN
only brings limited gain. In exploring a more effective approach, we find that
the key to a successful instance segmentation cascade is to fully leverage the
reciprocal relationship between detection and segmentation. In this work, we
propose a new framework, Hybrid Task Cascade (HTC), which differs in two
important aspects: (1) instead of performing cascaded refinement on these two
tasks separately, it interweaves them for a joint multi-stage processing; (2)
it adopts a fully convolutional branch to provide spatial context, which can
help distinguishing hard foreground from cluttered background. Overall, this
framework can learn more discriminative features progressively while
integrating complementary features together in each stage. Without bells and
whistles, a single HTC obtains 38.4 and 1.5 improvement over a strong Cascade
Mask R-CNN baseline on MSCOCO dataset. Moreover, our overall system achieves
48.6 mask AP on the test-challenge split, ranking 1st in the COCO 2018
Challenge Object Detection Task. Code is available at:
this https URL

提出了一种全新的混合任务级联（Hybrid Task Cascade，HTC）框架，基于实例分割级联的相互关系，在多个级别上逐步地学习可区分的特征并将互补特征整合在一起。相较于强基线，该方法在 MSCOCO 数据集上取得了 38.4 和 1.5 的提升，并在 COCO 2018 挑战目标检测任务中名列第一。