This paper presents an attempt to employ the mask language modeling approach
of BERT to pre-train a 12-layer Transformer model over 4,166 pieces of
polyphonic piano MIDI files for tackling a number of symbolic-domain
discriminative music understanding tasks. These include two note-level
classification tasks, i.e., melody extraction and velocity prediction, as well
as two sequence-level classification tasks, i.e., composer classification and
emotion classification. We find that, given a pre-trained Transformer, our
models outperform recurrent neural network based baselines with less than 10
epochs of fine-tuning. Ablation studies show that the pre-training remains
effective even if none of the MIDI data of the downstream tasks are seen at the
pre-training stage, and that freezing the self-attention layers of the
Transformer at the fine-tuning stage slightly degrades performance. All the
five datasets employed in this work are publicly available, as well as
checkpoints of our pre-trained and fine-tuned models. As such, our research can
be taken as a benchmark for symbolic-domain music understanding.

该研究使用 BERT 的掩蔽语言建模方法尝试预训练一个 12 层 Transformer 模型，用于处理 4166 个多音乐器 MIDI 文件，以解决一些基于符号领域的音乐理解任务，并发现预训练仍然有效。