We study speech-to-speech translation (S2ST) that translates speech from one
language into another language and focuses on building systems to support
languages without standard text writing systems. We use English-Taiwanese
Hokkien as a case study, and present an end-to-end solution from training data
collection, modeling choices to benchmark dataset release. First, we present
efforts on creating human annotated data, automatically mining data from large
unlabeled speech datasets, and adopting pseudo-labeling to produce weakly
supervised data. On the modeling, we take advantage of recent advances in
applying self-supervised discrete representations as target for prediction in
S2ST and show the effectiveness of leveraging additional text supervision from
Mandarin, a language similar to Hokkien, in model training. Finally, we release
an S2ST benchmark set to facilitate future research in this field. The demo can
be found at this https URL .

该论文以英文 - 台湾闽南语为案例，研究了语音到语音的翻译，包括数据收集、建模和基准数据集的发布。通过人工注释、数据自动挖掘以及伪标注方法，收集了大量数据。在模型训练中，结合最近的自监督离散表示的应用以及与福建话相似的普通话文本监督，取得了良好的效果。最后，该论文推出了一个 S2ST 基准数据集，以助于未来的研究。