Direct speech-to-speech translation (S2ST), in which all components can be
optimized jointly, is advantageous over cascaded approaches to achieve fast
inference with a simplified pipeline. We present a novel two-pass direct S2ST
architecture, {\textit UnitY}, which first generates textual representations
and predicts discrete acoustic units subsequently. We enhance the model
performance by subword prediction in the first-pass decoder, advanced two-pass
decoder architecture design and search strategy, and better training
regularization. To leverage large amounts of unlabeled text data, we pre-train
the first-pass text decoder based on the self-supervised denoising
auto-encoding task. Experimental evaluations on benchmark datasets at various
data scales demonstrate that UnitY outperforms a single-pass speech-to-unit
translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show
that the proposed methods boost the performance even when predicting
spectrogram in the second pass. However, predicting discrete units achieves
2.51x decoding speed-up compared to that case.

本文提出了一种名为 UnitY 的新型两遍直接 S2ST 架构，通过子词预测、先进的两遍解码器架构设计和搜索策略以及更好的训练规则等技术实现模型性能的提升，其在多个基准数据集上表现出比单遍语音转换模型高 2.5-4.2 ASR-BLEU 分数且同时具有 2.83 倍的解码速度提速。