We present a direct speech-to-speech translation (S2ST) model that translates
speech from one language to speech in another language without relying on
intermediate text generation. We tackle the problem by first applying a
self-supervised discrete speech encoder on the target speech and then training
a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the
discrete representations of the target speech. When target text transcripts are
available, we design a joint speech and text training framework that enables
the model to generate dual modality output (speech and text) simultaneously in
the same inference pass. Experiments on the Fisher Spanish-English dataset show
that the proposed framework yields improvement of 6.7 BLEU compared with a
baseline direct S2ST model that predicts spectrogram features. When trained
without any text transcripts, our model performance is comparable to models
that predict spectrograms and are trained with text supervision, showing the
potential of our system for translation between unwritten languages. Audio
samples are available at
this https URL .

本研究提出一种直接语音到语音的翻译模型，它可以在不依赖中间文本生成的情况下将一种语言的语音翻译成另一种语言的语音，并且结合了自我监督离散语音编码和序列到序列的语音到单位翻译，以预测目标语音的离散表示，并在同一推断通过实现同步生成双重模态输出（语音和文本）。在 Fisher 西班牙语 - 英语数据集上，我们的模型较基线直接 S2ST 模型改进了 6.7 BLEU。由于能够支持未书写语言的翻译，当没有文本转录时，我们的模型表现与训练有文本监督的谱图预测模型相当。

使用离散单元进行直接语音到语音翻译

Direct speech-to-speech translation with discrete units

Existing speech to speech translation systems heavily rely on the text of
target language: they usually translate source language either to target text
and then synthesize target speech from text, or directly to target speech with
target text for auxiliary training. However, those methods cannot be applied to
unwritten target languages, which have no written text or phoneme available. In
this paper, we develop a translation system for unwritten languages, named as
UWSpeech, which converts target unwritten speech into discrete tokens with a
converter, and then translates source-language speech into target discrete
tokens with a translator, and finally synthesizes target speech from target
discrete tokens with an inverter. We propose a method called XL-VAE, which
enhances vector quantized variational autoencoder (VQ-VAE) with cross-lingual
(XL) speech recognition, to train the converter and inverter of UWSpeech
jointly. Experiments on Fisher Spanish-English conversation translation dataset
show that UWSpeech outperforms direct translation and VQ-VAE baseline by about
16 and 10 BLEU points respectively, which demonstrate the advantages and
potentials of UWSpeech.

本文中，我们提出了一种名为 UWSpeech 的翻译系统，它将目标未书写语言转换为离散标记，然后用转换器将源语言语音转换为目标离散标记，最后使用倒置器从目标离散标记中合成目标语音。我们提出了一种名为 XL-VAE 的方法，用于共同训练 UWSpeech 的转换器和倒置器，并在 Fisher 西班牙语–英语交谈翻译数据集上进行了实验证明 UWSpeech 相较于直接翻译和 VQ-VAE 基线的表现提高了 16 和 10 BLEU 分数，证明了 UWSpeech 的优势和潜力。