With the increasing demand for video understanding, video moment and
highlight detection (MHD) has emerged as a critical research topic. MHD aims to
localize all moments and predict clip-wise saliency scores simultaneously.
Despite progress made by existing DETR-based methods, we observe that these
methods coarsely fuse features from different modalities, which weakens the
temporal intra-modal context and results in insufficient cross-modal
interaction. To address this issue, we propose MH-DETR (Moment and Highlight
Detection Transformer) tailored for MHD. Specifically, we introduce a simple
yet efficient pooling operator within the uni-modal encoder to capture global
intra-modal context. Moreover, to obtain temporally aligned cross-modal
features, we design a plug-and-play cross-modal interaction module between the
encoder and decoder, seamlessly integrating visual and textual features.
Comprehensive experiments on QVHighlights, Charades-STA, Activity-Net, and
TVSum datasets show that MH-DETR outperforms existing state-of-the-art methods,
demonstrating its effectiveness and superiority. Our code is available at
this https URL

提出了面向视频时刻和精彩片段检测的 MH-DETR 模型，其采用简单而高效的池化操作和跨模态交互模块，以获得更好的时序内模态上下文和跨模态特征。在多个数据集上的实验结果表明，MH-DETR 优于现有的最先进方法，具有较高的准确性和鲁棒性。