Identifying highlight moments of raw video materials is crucial for improving
the efficiency of editing videos that are pervasive on internet platforms.
However, the extensive work of manually labeling footage has created obstacles
to applying supervised methods to videos of unseen categories. The absence of
an audio modality that contains valuable cues for highlight detection in many
videos also makes it difficult to use multimodal strategies. In this paper, we
propose a novel model with cross-modal perception for unsupervised highlight
detection. The proposed model learns representations with visual-audio level
semantics from image-audio pair data via a self-reconstruction task. To achieve
unsupervised highlight detection, we investigate the latent representations of
the network and propose the representation activation sequence learning (RASL)
module with k-point contrastive learning to learn significant representation
activations. To connect the visual modality with the audio modality, we use the
symmetric contrastive learning (SCL) module to learn the paired visual and
audio representations. Furthermore, an auxiliary task of masked feature vector
sequence (FVS) reconstruction is simultaneously conducted during pretraining
for representation enhancement. During inference, the cross-modal pretrained
model can generate representations with paired visual-audio semantics given
only the visual modality. The RASL module is used to output the highlight
scores. The experimental results show that the proposed framework achieves
superior performance compared to other state-of-the-art approaches.

通过跨模态感知和自重建任务，在视觉 - 音频对数据中学习视觉 - 音频语义表示的网络，并通过表示激活序列学习模块（RASL）和对称对比学习模块（SCL）连接视觉模态与音频模态，提出了一种用于无监督高光检测的模型。在预训练期间，进行了掩码特征向量序列（FVS）重建的辅助任务，以增强表示。实验结果表明，所提出的框架相对于其他最先进的方法具有优越性能。