Large Language Models (LLMs) have demonstrated superior abilities in tasks
such as chatting, reasoning, and question-answering. However, standard LLMs may
ignore crucial paralinguistic information, such as sentiment, emotion, and
speaking style, which are essential for achieving natural, human-like spoken
conversation, especially when such information is conveyed by acoustic cues. We
therefore propose Paralinguistics-enhanced Generative Pretrained Transformer
(ParalinGPT), an LLM utilizes text and speech modality to better model the
linguistic content and paralinguistic attribute of spoken response. The model
takes the conversational context of text, speech embeddings, and paralinguistic
attributes as input prompts within a serialized multitasking multi-modal
framework. Specifically, our framework serializes tasks in the order of current
paralinguistic attribute prediction, response paralinguistic attribute
prediction, and response text generation with autoregressive conditioning. We
utilize the Switchboard-1 corpus, including its sentiment labels to be the
paralinguistic attribute, as our spoken dialogue dataset. Experimental results
indicate the proposed serialized multitasking method outperforms typical
sequence classification techniques on current and response sentiment
classification. Furthermore, leveraging conversational context and speech
embeddings significantly improves both response text generation and sentiment
prediction. Our proposed framework achieves relative improvements of 6.7%,
12.0%, and 3.5% in current sentiment accuracy, response sentiment accuracy, and
response text BLEU score, respectively.

大型语言模型在聊天、推理和问答等任务中表现出卓越的能力，然而标准的语言模型可能会忽略关键的语用信息，如情感、情感和口语风格，而这些信息对于实现自然、类似人类的口语对话非常重要，特别是当这些信息通过声学提示来传达时。因此，我们提出了一种增强语用的生成预训练变压器（ParalinGPT），该模型利用文本和语音模态来更好地建模口语回应的语言内容和语用属性。该模型将文本的对话背景、语音嵌入和语用属性作为输入提示，在序列化的多任务多模态框架中。实验结果表明，所提出的序列化多任务方法在当前和回应的情感分类上优于典型的序列分类技术。此外，利用对话背景和语音嵌入显著改进了回应文本的生成和情感预测。我们提出的框架在当前情感准确度、回应情感准确度和回应文本 BLEU 分数上分别取得了 6.7％、12.0％和 3.5％的相对改进。