There is a wide variety of speech processing tasks ranging from extracting
content information from speech signals to generating speech signals. For
different tasks, model networks are usually designed and tuned separately. If a
universal model can perform multiple speech processing tasks, some tasks might
be improved with the related abilities learned from other tasks. The multi-task
learning of a wide variety of speech processing tasks with a universal model
has not been studied. This paper proposes a universal modularized model,
SpeechNet, which treats all speech processing tasks into a speech/text input
and speech/text output format. We select five essential speech processing tasks
for multi-task learning experiments with SpeechNet. We show that SpeechNet
learns all of the above tasks, and we further analyze which tasks can be
improved by other tasks. SpeechNet is modularized and flexible for
incorporating more modules, tasks, or training approaches in the future. We
release the code and experimental settings to facilitate the research of
modularized universal models and multi-task learning of speech processing
tasks.

本文提出了一种通用模块化模型 SpeechNet，将所有语音处理任务分为语音 / 文本输入和语音 / 文本输出格式，并选择五个重要的语音处理任务进行多任务学习实验。研究表明，SpeechNet 学习了所有以上任务，并进一步分析了哪些任务可以通过其他任务得到改进。