Little research focuses on cross-modal correlation learning where temporal
structures of different data modalities such as audio and lyrics are taken into
account. Stemming from the characteristic of temporal structures of music in
nature, we are motivated to learn the deep sequential correlation between audio
and lyrics. In this work, we propose a deep cross-modal correlation learning
architecture involving two-branch deep neural networks for audio modality and
text modality (lyrics). Different modality data are converted to the same
canonical space where inter modal canonical correlation analysis is utilized as
an objective function to calculate the similarity of temporal structures. This
is the first study on understanding the correlation between language and music
audio through deep architectures for learning the paired temporal correlation
of audio and lyrics. Pre-trained Doc2vec model followed by fully-connected
layers (fully-connected deep neural network) is used to represent lyrics. Two
significant contributions are made in the audio branch, as follows: i)
pre-trained CNN followed by fully-connected layers is investigated for
representing music audio. ii) We further suggest an end-to-end architecture
that simultaneously trains convolutional layers and fully-connected layers to
better learn temporal structures of music audio. Particularly, our end-to-end
deep architecture contains two properties: simultaneously implementing feature
learning and cross-modal correlation learning, and learning joint
representation by considering temporal structures. Experimental results, using
audio to retrieve lyrics or using lyrics to retrieve audio, verify the
effectiveness of the proposed deep correlation learning architectures in
cross-modal music retrieval.

本文介绍了一种音频和歌词之间深度序列相关性的跨模态学习架构，其中涉及两个分支的深度神经网络，通过转换到相同的规范空间并利用互模态规范相关分析作为目标函数来计算时间结构的相似性。实验结果证实了所提出跨模态音乐检索的深度相关学习体系结构的有效性。