Recent advancements in generative models have sparked significant interest
within the machine learning community. Particularly, diffusion models have
demonstrated remarkable capabilities in synthesizing images and speech. Studies
such as those by Lee et al. [19], Black et al. [4], Wang et al. [36], and Fan
et al. [8] illustrate that Reinforcement Learning with Human Feedback (RLHF)
can enhance diffusion models for image synthesis. However, due to architectural
differences between these models and those employed in speech synthesis, it
remains uncertain whether RLHF could similarly benefit speech synthesis models.
In this paper, we explore the practical application of RLHF to diffusion-based
text-to-speech synthesis, leveraging the mean opinion score (MOS) as predicted
by UTokyo-SaruLab MOS prediction system [29] as a proxy loss. We introduce
diffusion model loss-guided RL policy optimization (DLPO) and compare it
against other RLHF approaches, employing the NISQA speech quality and
naturalness assessment model [21] and human preference experiments for further
evaluation. Our results show that RLHF can enhance diffusion-based
text-to-speech synthesis models, and, moreover, DLPO can better improve
diffusion models in generating natural and high quality speech audios.

利用强化学习和人类反馈进行扩散模型的文本转语音合成来生成自然且高质量的语音音频。