The goal of Automatic Voice Over (AVO) is to generate speech in sync with a
silent video given its text script. Recent AVO frameworks built upon
text-to-speech synthesis (TTS) have shown impressive results. However, the
current AVO learning objective of acoustic feature reconstruction brings in
indirect supervision for inter-modal alignment learning, thus limiting the
synchronization performance and synthetic speech quality. To this end, we
propose a novel AVO method leveraging the learning objective of self-supervised
discrete speech unit prediction, which not only provides more direct
supervision for the alignment learning, but also alleviates the mismatch
between the text-video context and acoustic features. Experimental results show
that our proposed method achieves remarkable lip-speech synchronization and
high speech quality by outperforming baselines in both objective and subjective
evaluations. Code and speech samples are publicly available.

该研究提出了一种基于自监督离散语音单元预测的 AVO 方法，该方法提供了更直接的对齐学习监督，改善了文本 - 视频上下文和声学特征之间的不匹配，实现了卓越的唇 - 语音同步和高品质语音合成效果。