Inspired by the insights in cognitive science with respect to human memory
and reasoning mechanism, a novel evolvable LLM-based (Large Language Model)
agent framework is proposed as REMEMBERER. By equipping the LLM with a
long-term experience memory, REMEMBERER is capable of exploiting the
experiences from the past episodes even for different task goals, which excels
an LLM-based agent with fixed exemplars or equipped with a transient working
memory. We further introduce Reinforcement Learning with Experience Memory
(RLEM) to update the memory. Thus, the whole system can learn from the
experiences of both success and failure, and evolve its capability without
fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER
constitutes a semi-parametric RL agent. Extensive experiments are conducted on
two RL task sets to evaluate the proposed framework. The average results with
different initialization and training sets exceed the prior SOTA by 4% and 2%
for the success rate on two task sets and demonstrate the superiority and
robustness of REMEMBERER.

提出一种新颖的可进化的 LLM（Large Language Model）智能体框架，称为 REMEMBERER，并将 LLM 配备了长期记忆，以便在不同的任务目标中利用过去的经验，这胜过具有固定示例或配备瞬态工作内存的 LLM 智能体。通过引入具有经验记忆的强化学习（RLEM）来更新记忆，整个系统可以从成功和失败的经验中学习，并在不微调 LLM 参数的情况下发展其能力。在两种强化学习任务集上进行了大量实验来评估所提出的框架，结果表明，通过不同的初始化和训练集，平均结果超过了先前的 SOTA 4％和 2％，证明了 REMEMBERER 的优越性和鲁棒性。