Conventional speech-to-text translation (ST) systems are trained on
single-speaker utterances, and they may not generalize to real-life scenarios
where the audio contains conversations by multiple speakers. In this paper, we
tackle single-channel multi-speaker conversational ST with an end-to-end and
multi-task training model, named Speaker-Turn Aware Conversational Speech
Translation, that combines automatic speech recognition, speech translation and
speaker turn detection using special tokens in a serialized labeling format. We
run experiments on the Fisher-CALLHOME corpus, which we adapted by merging the
two single-speaker channels into one multi-speaker channel, thus representing
the more realistic and challenging scenario with multi-speaker turns and
cross-talk. Experimental results across single- and multi-speaker conditions
and against conventional ST systems, show that our model outperforms the
reference systems on the multi-speaker condition, while attaining comparable
performance on the single-speaker condition. We release scripts for data
processing and model training.

本研究致力于解决单声道多说话者对话转写和翻译问题，提出了一种名为 Speaker-Turn Aware Conversational Speech Translation 的端到端多任务训练模型，通过在序列化标注格式中使用特殊标记来结合自动语音识别、语音翻译和说话者转换检测。在针对多说话者条件的实验中，我们使用合并了两个单声道通道的 Fisher-CALLHOME 语料库，以更真实和具有挑战性的场景来评估模型性能，实验结果表明我们的模型在多说话者条件下优于传统说话者转写系统，并且在单说话者条件下表现相当。同时，我们还提供了数据处理和模型训练的脚本。