Existing works on long-term open-domain dialogues focus on evaluating model
responses within contexts spanning no more than five chat sessions. Despite
advancements in long-context large language models (LLMs) and retrieval
augmented generation (RAG) techniques, their efficacy in very long-term
dialogues remains unexplored. To address this research gap, we introduce a
machine-human pipeline to generate high-quality, very long-term dialogues by
leveraging LLM-based agent architectures and grounding their dialogues on
personas and temporal event graphs. Moreover, we equip each agent with the
capability of sharing and reacting to images. The generated conversations are
verified and edited by human annotators for long-range consistency and
grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
dataset of very long-term conversations, each encompassing 300 turns and 9K
tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
comprehensive evaluation benchmark to measure long-term memory in models,
encompassing question answering, event summarization, and multi-modal dialogue
generation tasks. Our experimental results indicate that LLMs exhibit
challenges in understanding lengthy conversations and comprehending long-range
temporal and causal dynamics within dialogues. Employing strategies like
long-context LLMs or RAG can offer improvements but these models still
substantially lag behind human performance.

通过使用基于 LLM 的代理体系结构和将对话与人物和时间事件图进行关联，我们介绍了一个机器 - 人类管道来生成高质量的非常长期的对话，并通过人类注释者对其进行检验和编辑，以确保其长程一致性和对事件图的关联。通过这个管道，我们收集了一个包含 300 个回合和平均 9K 个记号的非常长期对话的数据集。基于该数据集，我们提出了一个全面的评估基准来衡量模型中的长期记忆，在问题回答、事件摘要和多模态对话生成任务方面。我们的实验结果表明，LLM 在理解冗长对话和理解对话中的长程时间和因果动态方面存在挑战。使用长上下文 LLM 或 RAG 等策略可以提供改进，但这些模型仍然远远落后于人类的性能。