We propose PolyVoice, a language model-based framework for speech-to-speech
translation (S2ST) system. Our framework consists of two language models: a
translation language model and a speech synthesis language model. We use
discretized speech units, which are generated in a fully unsupervised way, and
thus our framework can be used for unwritten languages. For the speech
synthesis part, we adopt the existing VALL-E X approach and build a unit-based
audio language model. This grants our framework the ability to preserve the
voice characteristics and the speaking style of the original speech. We examine
our system on Chinese $\rightarrow$ English and English $\rightarrow$ Spanish
pairs. Experimental results show that our system can generate speech with high
translation quality and audio quality. Speech samples are available at
this https URL

我们提出了 PolyVoice，这是一个基于语言模型的语音 - 语音翻译（S2ST）系统，利用离散化语音单元实现对未书写语言的翻译。在语音合成方面，我们采用现有的 VALL-E X 方法构建基于单元的音频语言模型来保留原始语音的音色特征和语音风格。实验结果表明，我们的系统可以生成高质量翻译和音频质量的语音。