Self-supervised learning (SSL) has shown promising results in various speech
and natural language processing applications. However, its efficacy in music
information retrieval (MIR) still remains largely unexplored. While previous
SSL models pre-trained on music recordings may have been mostly closed-sourced,
recent speech models such as wav2vec2.0 have shown promise in music modelling.
Nevertheless, research exploring the effectiveness of applying speech SSL
models to music recordings has been limited. We explore the music adaption of
SSL with two distinctive speech-related models, data2vec1.0 and Hubert, and
refer to them as music2vec and musicHuBERT, respectively. We train $12$ SSL
models with 95M parameters under various pre-training configurations and
systematically evaluate the MIR task performances with 13 different MIR tasks.
Our findings suggest that training with music data can generally improve
performance on MIR tasks, even when models are trained using paradigms designed
for speech. However, we identify the limitations of such existing
speech-oriented designs, especially in modelling polyphonic information. Based
on the experimental results, empirical suggestions are also given for designing
future musical SSL strategies and paradigms.

通过使用两个音频方案 data2vec1.0 和 Hubert，我们训练了 $12$ 个自监督学习模型，并在 13 个不同的音乐信息检索任务中进行系统评估。我们的研究表明，使用音乐数据进行训练可以提高 MIR 任务性能，即使使用的是设计用于语音的模型，也能取得效果。但是，我们还发现现有的语音定向设计存在一些局限性，特别是在多声部信息建模方面。因此，我们为未来的音乐自监督学习策略和范式提供了实证建议。