Existing speech to speech translation systems heavily rely on the text of
target language: they usually translate source language either to target text
and then synthesize target speech from text, or directly to target speech with
target text for auxiliary training. However, those methods cannot be applied to
unwritten target languages, which have no written text or phoneme available. In
this paper, we develop a translation system for unwritten languages, named as
UWSpeech, which converts target unwritten speech into discrete tokens with a
converter, and then translates source-language speech into target discrete
tokens with a translator, and finally synthesizes target speech from target
discrete tokens with an inverter. We propose a method called XL-VAE, which
enhances vector quantized variational autoencoder (VQ-VAE) with cross-lingual
(XL) speech recognition, to train the converter and inverter of UWSpeech
jointly. Experiments on Fisher Spanish-English conversation translation dataset
show that UWSpeech outperforms direct translation and VQ-VAE baseline by about
16 and 10 BLEU points respectively, which demonstrate the advantages and
potentials of UWSpeech.

本文中，我们提出了一种名为 UWSpeech 的翻译系统，它将目标未书写语言转换为离散标记，然后用转换器将源语言语音转换为目标离散标记，最后使用倒置器从目标离散标记中合成目标语音。我们提出了一种名为 XL-VAE 的方法，用于共同训练 UWSpeech 的转换器和倒置器，并在 Fisher 西班牙语–英语交谈翻译数据集上进行了实验证明 UWSpeech 相较于直接翻译和 VQ-VAE 基线的表现提高了 16 和 10 BLEU 分数，证明了 UWSpeech 的优势和潜力。