It has been known that direct speech-to-speech translation (S2ST) models
usually suffer from the data scarcity issue because of the limited existing
parallel materials for both source and target speech. Therefore to train a
direct S2ST system, previous works usually utilize text-to-speech (TTS) systems
to generate samples in the target language by augmenting the data from
speech-to-text translation (S2TT). However, there is a limited investigation
into how the synthesized target speech would affect the S2ST models. In this
work, we analyze the effect of changing synthesized target speech for direct
S2ST models. We find that simply combining the target speech from different TTS
systems can potentially improve the S2ST performances. Following that, we also
propose a multi-task framework that jointly optimizes the S2ST system with
multiple targets from different TTS systems. Extensive experiments demonstrate
that our proposed framework achieves consistent improvements (2.8 BLEU) over
the baselines on the Fisher Spanish-English dataset.

通过分析不同合成目标语音对直接语音翻译模型的影响，本文提出了一个多任务框架，通过同时优化不同 TTS 系统的多个目标来提高 S2ST 表现，在 Fisher 西班牙语 - 英语数据集上实现了 2.8 BLEU 的一致改善效果。