Multimodal language models that process both text and speech have a potential
for applications in spoken dialogue systems. However, current models face two
major challenges in response generation latency: (1) generating a spoken
response requires the prior generation of a written response, and (2) speech
sequences are significantly longer than text sequences. This study addresses
these issues by extending the input and output sequences of the language model
to support the parallel generation of text and speech. Our experiments on
spoken question answering tasks demonstrate that our approach improves latency
while maintaining the quality of response content. Additionally, we show that
latency can be further reduced by generating speech in multiple sequences. Demo
samples are available at this https URL

多模式语言模型通过扩展输入和输出序列，支持并行生成文本和语音，从而改善响应生成的延迟，并保持响应内容的质量。在口语问答任务上的实验证明，该方法进一步通过生成多个语音序列来减少延迟。