This work addresses the problem of matching short excerpts of audio with
their respective counterparts in sheet music images. We show how to employ
neural network-based cross-modality embedding spaces for solving the following
two sheet music-related tasks: retrieving the correct piece of sheet music from
a database when given a music audio as a search query; and aligning an audio
recording of a piece with the corresponding images of sheet music. We
demonstrate the feasibility of this in experiments on classical piano music by
five different composers (Bach, Haydn, Mozart, Beethoven and Chopin), and
additionally provide a discussion on why we expect multi-modal neural networks
to be a fruitful paradigm for dealing with sheet music and audio at the same
time.

该研究使用神经网络跨模态嵌入空间解决音频和乐谱图像之间的匹配问题，并在五个不同作曲家的古典钢琴音乐上进行实验，展示出多模态神经网络处理乐谱和音频的成果。

针对乐谱识别和离线对齐的音频 - 乐谱对应关系学习

Learning Audio - Sheet Music Correspondences for Score Identification  and Offline Alignment

This paper addresses the matching of short music audio snippets to the
corresponding pixel location in images of sheet music. A system is presented
that simultaneously learns to read notes, listens to music and matches the
currently played music to its corresponding notes in the sheet. It consists of
an end-to-end multi-modal convolutional neural network that takes as input
images of sheet music and spectrograms of the respective audio snippets. It
learns to predict, for a given unseen audio snippet (covering approximately one
bar of music), the corresponding position in the respective score line. Our
results suggest that with the use of (deep) neural networks -- which have
proven to be powerful image processing models -- working with sheet music
becomes feasible and a promising future research direction.

本文介绍了一种通过神经网络匹配短音频片段与乐谱中对应像素位置的方法。实验结果表明，使用深度学习技术处理音乐乐谱具有可行性且是一个有前途的研究方向。