Recently, there have been attempts to integrate various speech processing
tasks into a unified model. However, few previous works directly demonstrated
that joint optimization of diverse tasks in multitask speech models has
positive influence on the performance of individual tasks. In this paper we
present a multitask speech model -- PolySpeech, which supports speech
recognition, speech synthesis, and two speech classification tasks. PolySpeech
takes multi-modal language model as its core structure and uses semantic
representations as speech inputs. We introduce semantic speech embedding
tokenization and speech reconstruction methods to PolySpeech, enabling
efficient generation of high-quality speech for any given speaker. PolySpeech
shows competitiveness across various tasks compared to single-task models. In
our experiments, multitask optimization achieves performance comparable to
single-task optimization and is especially beneficial for specific tasks.

该研究提出了一种多任务语音模型 (PolySpeech)，支持语音识别、语音合成和两个语音分类任务。PolySpeech 通过引入多模态语言模型作为核心结构，并使用语义表示作为语音输入，实现了多模态优化。实验结果表明，多任务优化在特定任务上表现出可与单任务优化相比的性能，并在各种任务中显示出与单任务模型相媲美的竞争力。