Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech
translation) outputs target speech while receiving streaming speech inputs,
which is critical for real-time communication. Beyond accomplishing translation
between speech, Simul-S2ST requires a policy to control the model to generate
corresponding target speech at the opportune moment within speech inputs,
thereby posing a double challenge of translation and policy. In this paper, we
propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation
and simultaneous policy in a unified framework of multi-task learning. Adhering
to a multi-task learning approach, StreamSpeech can perform offline and
simultaneous speech recognition, speech translation and speech synthesis via an
"All-in-One" seamless model. Experiments on CVSS benchmark demonstrate that
StreamSpeech achieves state-of-the-art performance in both offline S2ST and
Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality
intermediate results (i.e., ASR or translation results) during simultaneous
translation process, offering a more comprehensive real-time communication
experience.

Simultaneous speech-to-speech translation (Simul-S2ST) is achieved through StreamSpeech, a unified model that incorporates translation and simultaneous policy in multi-task learning, providing high-quality intermediate results and a comprehensive real-time communication experience.