Multimodal and large language models (LLMs) have revolutionized the
utilization of open-world knowledge, unlocking novel potentials across various
tasks and applications. Among these domains, the video domain has notably
benefited from their capabilities. In this paper, we present Highlight-CLIP
(HL-CLIP), a method designed to excel in the video highlight detection task by
leveraging the pre-trained knowledge embedded in multimodal models. By simply
fine-tuning the multimodal encoder in combination with our innovative saliency
pooling technique, we have achieved the state-of-the-art performance in the
highlight detection task, the QVHighlight Benchmark, to the best of our
knowledge.

通过利用多模态模型中预训练知识和创新的显著性池化技术，我们在视频亮点检测任务中通过简单微调多模态编码器达到了最先进的性能，对于 QVHighlight 基准测试而言，可获得最佳表现。

释放 CLIP 在视频亮点检测中的潜力

Unleash the Potential of CLIP for Video Highlight Detection

The goal of video highlight detection is to select the most attractive
segments from a long video to depict the most interesting parts of the video.
Existing methods typically focus on modeling relationship between different
video segments in order to learning a model that can assign highlight scores to
these segments; however, these approaches do not explicitly consider the
contextual dependency within individual segments. To this end, we propose to
learn pixel-level distinctions to improve the video highlight detection. This
pixel-level distinction indicates whether or not each pixel in one video
belongs to an interesting section. The advantages of modeling such fine-level
distinctions are two-fold. First, it allows us to exploit the temporal and
spatial relations of the content in one video, since the distinction of a pixel
in one frame is highly dependent on both the content before this frame and the
content around this pixel in this frame. Second, learning the pixel-level
distinction also gives a good explanation to the video highlight task regarding
what contents in a highlight segment will be attractive to people. We design an
encoder-decoder network to estimate the pixel-level distinction, in which we
leverage the 3D convolutional neural networks to exploit the temporal context
information, and further take advantage of the visual saliency to model the
spatial distinction. State-of-the-art performance on three public benchmarks
clearly validates the effectiveness of our framework for video highlight
detection.

本文提出了一种基于像素级别特征的视频亮点检测方法，通过学习每个像素在视频中的所属类别，既考虑了时间序列特征，也考虑了空间特征；并且通过使用 3D 卷积神经网络和视觉显著性模型，构建了一个编码 - 解码网络，在三个公共基准测试集上均取得了最先进的效果。

学习像素级别的视频精华检测

Learning Pixel-Level Distinctions for Video Highlight Detection

We address the weakly supervised video highlight detection problem for
learning to detect segments that are more attractive in training videos given
their video event label but without expensive supervision of manually
annotating highlight segments. While manually averting localizing highlight
segments, weakly supervised modeling is challenging, as a video in our daily
life could contain highlight segments with multiple event types, e.g., skiing
and surfing. In this work, we propose casting weakly supervised video highlight
detection modeling for a given specific event as a multiple instance ranking
network (MINI-Net) learning. We consider each video as a bag of segments, and
therefore, the proposed MINI-Net learns to enforce a higher highlight score for
a positive bag that contains highlight segments of a specific event than those
for negative bags that are irrelevant. In particular, we form a max-max ranking
loss to acquire a reliable relative comparison between the most likely positive
segment instance and the hardest negative segment instance. With this max-max
ranking loss, our MINI-Net effectively leverages all segment information to
acquire a more distinct video feature representation for localizing the
highlight segments of a specific event in a video. The extensive experimental
results on three challenging public benchmarks clearly validate the efficacy of
our multiple instance ranking approach for solving the problem.

本文提出了一种弱监督视频亮点检测建模方法，将其作为多实例排名网络（MINI-Net）学习，并使用 max-max 排名损失以获取可靠的相对比较，实现在视频中定位特定事件的高亮片段，实验证明其在三大公共基准测试中的有效性。