In spoken dialogue, even if two current turns are the same sentence, their
responses might still differ when they are spoken in different styles. The
spoken styles, containing paralinguistic and prosodic information, mark the
most significant difference between text and speech modality. When using
text-only LLMs to model spoken dialogue, text-only LLMs cannot give different
responses based on the speaking style of the current turn. In this paper, we
focus on enabling LLMs to listen to the speaking styles and respond properly.
Our goal is to teach the LLM that "even if the sentences are identical if they
are spoken in different styles, their corresponding responses might be
different". Since there is no suitable dataset for achieving this goal, we
collect a speech-to-speech dataset, StyleTalk, with the following desired
characteristics: when two current speeches have the same content but are spoken
in different styles, their responses will be different. To teach LLMs to
understand and respond properly to the speaking styles, we propose the
Spoken-LLM framework that can model the linguistic content and the speaking
styles. We train Spoken-LLM using the StyleTalk dataset and devise a two-stage
training pipeline to help the Spoken-LLM better learn the speaking styles.
Based on extensive experiments, we show that Spoken-LLM outperforms text-only
baselines and prior speech LLMs methods.

在口语对话中，即使两个当前对话相同，它们的回应在不同的语言风格中可能仍然不同。这篇论文提出了一种 Spoken-LLM 框架，旨在教导 LLMs 理解和适当回应不同的语言风格，并使用 StyleTalk 数据集进行训练，通过两个阶段的训练使 Spoken-LLM 更好地学习语言风格，实验证明 Spoken-LLM 表现优于纯文本基准和先前的语音 LLMs 方法。