We present an approach for recommending a music track for a given video, and
vice versa, based on both their temporal alignment and their correspondence at
an artistic level. We propose a self-supervised approach that learns this
correspondence directly from data, without any need of human annotations. In
order to capture the high-level concepts that are required to solve the task,
we propose modeling the long-term temporal context of both the video and the
music signals, using Transformer networks for each modality. Experiments show
that this approach strongly outperforms alternatives that do not exploit the
temporal context. The combination of our contributions improve retrieval
accuracy up to 10x over prior state of the art. This strong improvement allows
us to introduce a wide range of analyses and applications. For instance, we can
condition music retrieval based on visually defined attributes.

本文提出了一种基于自我监督学习的方法，使用 Transformer 网络建模视频和音乐信号的长期时间上下文，以实现音乐和视频之间的艺术语义对应以及它们之间的互相推荐，并在实验中表现出很强的性能优势。