Large language model (LLM)-based decision-making agents have shown the
ability to generalize across multiple tasks. However, their performance relies
on massive data and compute. We argue that this inefficiency stems from the
forgetting phenomenon, in which a model memorizes its behaviors in parameters
throughout training. As a result, training on a new task may deteriorate the
model's performance on previous tasks. In contrast to LLMs' implicit memory
mechanism, the human brain utilizes distributed memory storage, which helps
manage and organize multiple skills efficiently, mitigating the forgetting
phenomenon. Thus inspired, we propose an internal working memory module to
store, blend, and retrieve information for different downstream tasks.
Evaluation results show that the proposed method improves training efficiency
and generalization in both Atari games and meta-world object manipulation
tasks. Moreover, we demonstrate that memory fine-tuning further enhances the
adaptability of the proposed architecture.

本文提出了一种基于内部工作记忆模块的决策制定代理，可以通过存储、混合和检索信息来改善其在不同下游任务中的训练效率和泛化能力，并进一步证明记忆微调可以增强所提出架构的适应性。

深思熟虑：带有内部工作记忆的决策 Transformer

Think Before You Act: Decision Transformers with Internal Working Memory

In this work, we study how the finetuning stage in the pretrain-finetune
framework changes the behavior of a pretrained neural language generator. We
focus on the transformer encoder-decoder model for the open-domain dialogue
response generation task. Our major finding is that after standard finetuning,
the model forgets some of the important language generation skills acquired
during large-scale pretraining. We demonstrate the forgetting phenomenon
through a set of detailed behavior analysis from the perspectives of knowledge
transfer, context sensitivity, and function space projection. As a preliminary
attempt to alleviate the forgetting problem, we propose an intuitive finetuning
strategy named "mix-review". We find that mix-review effectively regularizes
the finetuning process, and the forgetting problem is alleviated to some
extent. Finally, we discuss interesting behavior of the resulting dialogue
model and its implications.

本研究探讨预训练 - 微调框架中的微调阶段如何改变预训练神经语言生成模型的行为，发现标准微调后模型遗忘了某些重要的语言生成技巧，提出了一种直观的微调策略 “混合回顾”，能够缓解模型遗忘现象，最终讨论了对话模型的有趣行为及其含义。