LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim.

本研究解决了当前缺乏对基于LLM的个人助理记忆能力进行客观自动评估的问题。通过提出MemSim，这一创新的贝叶斯模拟器能够自动构建用户消息生成的可靠问答，显著提升评估数据集的多样性和可扩展性。研究表明，MemSim不仅可以生成日常生活场景的数据集MemDaily，还为不同记忆机制的评估提供了基准，具有重要的影响力。

MemSim：用于评估基于LLM的个人助理记忆能力的贝叶斯模拟器