Up to now, only limited research has been conducted on cross-modal retrieval
of suitable music for a specified video or vice versa. Moreover, much of the
existing research relies on metadata such as keywords, tags, or associated
description that must be individually produced and attached posterior. This
paper introduces a new content-based, cross-modal retrieval method for video
and music that is implemented through deep neural networks. We train the
network via inter-modal ranking loss such that videos and music with similar
semantics end up close together in the embedding space. However, if only the
inter-modal ranking constraint is used for embedding, modality-specific
characteristics can be lost. To address this problem, we propose a novel soft
intra-modal structure loss that leverages the relative distance relationship
between intra-modal samples before embedding. We also introduce reasonable
quantitative and qualitative experimental protocols to solve the lack of
standard protocols for less-mature video-music related tasks. Finally, we
construct a large-scale 200K video-music pair benchmark. All the datasets and
source code can be found in our online repository
(this https URL).

本文介绍了一种新的基于内容的跨模态检索方法，通过深度神经网络实现视频和音乐之间的交叉检索，并使用软内部模态结构损失来保留模态特征，同时提出了标准化的实验协议并构建了一个大规模的视频 - 音乐匹配基准。