As a subjective metric to evaluate the quality of synthesized speech, Mean
opinion score~(MOS) usually requires multiple annotators to score the same
speech. Such an annotation approach requires a lot of manpower and is also
time-consuming. MOS prediction model for automatic evaluation can significantly
reduce labor cost. In previous works, it is difficult to accurately rank the
quality of speech when the MOS scores are close. However, in practical
applications, it is more important to correctly rank the quality of synthesis
systems or sentences than simply predicting MOS scores. Meanwhile, as each
annotator scores multiple audios during annotation, the score is probably a
relative value based on the first or the first few speech scores given by the
annotator. Motivated by the above two points, we propose a general framework
for MOS prediction based on pair comparison (MOSPC), and we utilize C-Mixup
algorithm to enhance the generalization performance of MOSPC. The experiments
on BVCC and VCC2018 show that our framework outperforms the baselines on most
of the correlation coefficient metrics, especially on the metric KTAU related
to quality ranking. And our framework also surpasses the strong baseline in
ranking accuracy on each fine-grained segment. These results indicate that our
framework contributes to improving the ranking accuracy of speech quality.

提出了基于配对比较的 MOS 预测框架，并利用 C-Mixup 算法增强其泛化性能，实验结果表明该框架在语音质量的排名准确性上优于当前基准，说明该框架对于提高语音质量排名的准确性具有改进作用。

基于两两比较的 MOS 预测

MOSPC: MOS Prediction Based on Pairwise Comparison

We train a MOS prediction model based on wav2vec 2.0 using the open-access
data sets BVCC and SOMOS. Our test with neural TTS data in the low-resource
language (LRL) West Frisian shows that pre-training on BVCC before fine-tuning
on SOMOS leads to the best accuracy for both fine-tuned and zero-shot
prediction. Further fine-tuning experiments show that using more than 30
percent of the total data does not lead to significant improvements. In
addition, fine-tuning with data from a single listener shows promising
system-level accuracy, supporting the viability of one-participant pilot tests.
These findings can all assist the resource-conscious development of TTS for
LRLs by progressing towards better zero-shot MOS prediction and informing the
design of listening tests, especially in early-stage evaluation.

通过对 BVCC 和 SOMOS 数据集进行 wav2vec 2.0 预训练和微调，实现了针对 LRL West Frisian 的零样本 MOS 预测。研究表明，在 30% 的数据使用量之后，使用更多的数据并没有显著性的提高性能；同时，对单个听众的数据进行微调有望提高系统级的性能和稳定性。这些发现有助于开发 LRL 语音合成系统的零样本 MOS 预测和初期评估。