Standard Large Language Models (LLMs) struggle with handling dialogues with
long contexts due to efficiency and consistency issues. According to our
observation, dialogue contexts are highly structured, and the special token of
\textit{End-of-Utterance} (EoU) in dialogues has the potential to aggregate
information. We refer to the EoU tokens as ``conversational attention sinks''
(conv-attn sinks). Accordingly, we introduce StreamingDialogue, which
compresses long dialogue history into conv-attn sinks with minimal losses, and
thus reduces computational complexity quadratically with the number of sinks
(i.e., the number of utterances). Current LLMs already demonstrate the ability
to handle long context window, e.g., a window size of 200k or more. To this
end, by compressing utterances into EoUs, our method has the potential to
handle more than 200k of utterances, resulting in a prolonged dialogue
learning. In order to minimize information losses from reconstruction after
compression, we design two learning strategies of short-memory reconstruction
(SMR) and long-memory reactivation (LMR). Our method outperforms strong
baselines in dialogue tasks and achieves a 4 $\times$ speedup while reducing
memory usage by 18 $\times$ compared to dense attention recomputation.

通过将长对话历史压缩为最少损失的会话关注点（conv-attn sinks），我们的方法大幅减少了计算复杂度，并分别设计了短期记忆重构（SMR）和长期记忆激活（LMR）的学习策略，使得该方法在对话任务中优于强基线，并实现了 4 倍加速和 18 倍内存使用减少。

流式对话：通过最小损失的长上下文压缩实现长时间对话学习

StreamingDialogue: Prolonged Dialogue Learning via Long Context  Compression with Minimal Losses

Knowledge-grounded dialogue systems aim to generate coherent and engaging
responses based on the dialogue contexts and selected external knowledge.
Previous knowledge selection methods tend to rely too heavily on the dialogue
contexts or over-emphasize the new information in the selected knowledge,
resulting in the selection of repetitious or incongruous knowledge and further
generating repetitive or incoherent responses, as the generation of the
response depends on the chosen knowledge. To address these shortcomings, we
introduce a Coherent and Engaging Topic Transition (CET2) framework to model
topic transitions for selecting knowledge that is coherent to the context of
the conversations while providing adequate knowledge diversity for topic
development. Our CET2 framework considers multiple factors for knowledge
selection, including valid transition logic from dialogue contexts to the
following topics and systematic comparisons between available knowledge
candidates. Extensive experiments on two public benchmarks demonstrate the
superiority and the better generalization ability of CET2 on knowledge
selection. This is due to our well-designed transition features and comparative
knowledge selection strategy, which are more transferable to conversations
about unseen topics. Analysis of fine-grained knowledge selection accuracy also
shows that CET2 can better balance topic entailment (contextual coherence) and
development (knowledge diversity) in dialogue than existing approaches.

基于对话上下文和选定的外部知识生成一致且吸引人的响应的知识驱动对话系统，引入了一种具有连贯性和吸引力的主题转换（CET2）框架，以模拟选择与对话上下文一致且提供充足的主题发展知识的主题转换。CET2 框架考虑了多个因素，包括从对话上下文到下一个主题的有效转换逻辑和可用知识候选项之间的系统比较。在两个公共基准测试上的广泛实验表明，CET2 在知识选择方面具有更好的优势和更好的泛化能力。精细粒度知识选择准确性的分析还表明，CET2 可以在对话中更好地平衡主题蕴含（上下文连贯性）和发展（知识多样性），优于现有方法。

CET2: 模拟主题转换以实现连贯和引人入胜的基于知识的对话

CET2: Modelling Topic Transitions for Coherent and Engaging  Knowledge-Grounded Conversations

Dialogue contexts are proven helpful in the spoken language understanding
(SLU) system and they are typically encoded with explicit memory
representations. However, most of the previous models learn the context memory
with only one objective to maximizing the SLU performance, leaving the context
memory under-exploited. In this paper, we propose a new dialogue logistic
inference (DLI) task to consolidate the context memory jointly with SLU in the
multi-task framework. DLI is defined as sorting a shuffled dialogue session
into its original logical order and shares the same memory encoder and
retrieval mechanism as the SLU model. Our experimental results show that
various popular contextual SLU models can benefit from our approach, and
improvements are quite impressive, especially in slot filling.

本研究提出了一种新的对话逻辑推理（DLI）任务，在多任务框架中共同巩固上下文记忆和 SLU 模型，实验结果表明各种领先的上下文 SLU 模型可以从我们的方法中获益，尤其是在槽填充方面。