In this work, we present the SOMOS dataset, the first large-scale mean
opinion scores (MOS) dataset consisting of solely neural text-to-speech (TTS)
samples. It can be employed to train automatic MOS prediction systems focused
on the assessment of modern synthesizers, and can stimulate advancements in
acoustic model evaluation. It consists of 20K synthetic utterances of the LJ
Speech voice, a public domain speech dataset which is a common benchmark for
building neural acoustic models and vocoders. Utterances are generated from 200
TTS systems including vanilla neural acoustic models as well as models which
allow prosodic variations. An LPCNet vocoder is used for all systems, so that
the samples' variation depends only on the acoustic models. The synthesized
utterances provide balanced and adequate domain and length coverage. We collect
MOS naturalness evaluations on 3 English Amazon Mechanical Turk locales and
share practices leading to reliable crowdsourced annotations for this task. We
provide baseline results of state-of-the-art MOS prediction models on the SOMOS
dataset and show the limitations that such models face when assigned to
evaluate TTS utterances.

该论文介绍了一个大型的神经文本转语音（TTS）数据集 SOMOS，该数据集可以用来训练自动 MOS 预测系统，以评估现代合成器。该数据集由 20k 个来自 200 种 TTS 系统的合成语音组成，提供平衡和充足的领域和长度覆盖率，并在三个英文 Amazon Mechanical Turk 地点上收集 MOS 自然度评估，同时分享了可靠的众包注释实践。研究者提供了 SOTA MOS 预测模型在 SOMOS 数据集上的基线结果，并显示了这些模型在评估 TTS 语音时面临的限制。