Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.

本研究解决了对话语音合成中缺乏对话上下文和情感识别的问题。提出的JELLY框架通过集成情感识别和上下文推理，利用多个部分LoRA模块对大型语言模型进行微调，以生成合适的对话语音。实验结果表明，JELLY在情感上下文建模方面表现出色，能够自然地生成符合对话情感的语音。

JELLY: 结合情感识别与上下文推理的大型语言模型对话语音合成