Image is usually taken for expressing some kinds of emotions or purposes,
such as love, celebrating Christmas. There is another better way that combines
the image and relevant song to amplify the expression, which has drawn much
attention in the social network recently. Hence, the automatic selection of
songs should be expected. In this paper, we propose to retrieve semantic
relevant songs just by an image query, which is named as the image2song
problem. Motivated by the requirements of establishing correlation in
semantic/content, we build a semantic-based song retrieval framework, which
learns the correlation between image content and lyric words. This model uses a
convolutional neural network to generate rich tags from image regions, a
recurrent neural network to model lyric, and then establishes correlation via a
multi-layer perceptron. To reduce the content gap between image and lyric, we
propose to make the lyric modeling focus on the main image content via a tag
attention. We collect a dataset from the social-sharing multimodal data to
study the proposed problem, which consists of (image, music clip, lyric)
triplets. We demonstrate that our proposed model shows noticeable results in
the image2song retrieval task and provides suitable songs. Besides, the
song2image task is also performed.

本文提出一种基于语义的图像检索语音的解决方案，通过深度学习模型生成标签，使用神经网络建模歌词，并基于多层感知器建立图像和歌词之间的语义相关性，实验结果显示出明显的检索表现力。