Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech
translation) outputs target speech while receiving streaming speech inputs,
which is critical for real-time communication. Beyond accomplishing translation
between speech, Simul-S2ST requires a policy to control the model to generate
corresponding target speech at the opportune moment within speech inputs,
thereby posing a double challenge of translation and policy. In this paper, we
propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation
and simultaneous policy in a unified framework of multi-task learning. Adhering
to a multi-task learning approach, StreamSpeech can perform offline and
simultaneous speech recognition, speech translation and speech synthesis via an
"All-in-One" seamless model. Experiments on CVSS benchmark demonstrate that
StreamSpeech achieves state-of-the-art performance in both offline S2ST and
Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality
intermediate results (i.e., ASR or translation results) during simultaneous
translation process, offering a more comprehensive real-time communication
experience.

Simultaneous speech-to-speech translation (Simul-S2ST) is achieved through StreamSpeech, a unified model that incorporates translation and simultaneous policy in multi-task learning, providing high-quality intermediate results and a comprehensive real-time communication experience.

流式语音：多任务学习与同时语音翻译

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task  Learning

A popular approach to streaming speech translation is to employ a single
offline model with a \textit{wait-$k$} policy to support different latency
requirements, which is simpler than training multiple online models with
different latency constraints. However, there is a mismatch problem in using a
model trained with complete utterances for streaming inference with partial
input. We demonstrate that speech representations extracted at the end of a
streaming input are significantly different from those extracted from a
complete utterance. To address this issue, we propose a new approach called
Future-Aware Streaming Translation (FAST) that adapts an offline ST model for
streaming input. FAST includes a Future-Aware Inference (FAI) strategy that
incorporates future context through a trainable masked embedding, and a
Future-Aware Distillation (FAD) framework that transfers future context from an
approximation of full speech to streaming input. Our experiments on the MuST-C
EnDe, EnEs, and EnFr benchmarks show that FAST achieves better trade-offs
between translation quality and latency than strong baselines. Extensive
analyses suggest that our methods effectively alleviate the aforementioned
mismatch problem between offline training and online inference.

我们提出了一种名为 FAST 的方法，它可以为流输入调整离线 ST 模型，包括将未来信息集成到推理策略中和将未来信息从全语音的近似传输到流输入的框架，从而解决了离线训练和在线推理之间的不匹配问题，并在 MuST-C EnDe，EnEs 和 EnFr 基准测试中比强基线模型实现更好的平衡翻译质量和延迟。