Recent research on instructable agents has used memory-augmented Large
Language Models (LLMs) as task planners, a technique that retrieves
language-program examples relevant to the input instruction and uses them as
in-context examples in the LLM prompt to improve the performance of the LLM in
inferring the correct action and task plans. In this technical report, we
extend the capabilities of HELPER, by expanding its memory with a wider array
of examples and prompts, and by integrating additional APIs for asking
questions. This simple expansion of HELPER into a shared memory enables the
agent to work across the domains of executing plans from dialogue, natural
language instruction following, active question asking, and commonsense room
reorganization. We evaluate the agent on four diverse interactive
visual-language embodied agent benchmarks: ALFRED, TEACh, DialFRED, and the
Tidy Task. HELPER-X achieves few-shot, state-of-the-art performance across
these benchmarks using a single agent, without requiring in-domain training,
and remains competitive with agents that have undergone in-domain training.

扩展 HELPER 的记忆范围，整合额外的 API 来提问，使得代理可以跨多个领域进行计划执行、自然语言指令跟踪、主动提问和常识房间重组，HELPER-X 在四个不同的互动视觉 - 语言体验代理基准测试中取得了少量训练和最新框架表现，与接受过特定领域培训的代理保持一定的竞争力。

HELPER-X：基于记忆增强语言模型的统一可指导体现代理人解决四个交互式视觉语言领域

HELPER-X: A Unified Instructable Embodied Agent to Tackle Four  Interactive Vision-Language Domains with Memory-Augmented Language Models

Memory-augmented Large Language Models (LLMs) have demonstrated remarkable
performance in long-term human-machine interactions, which basically relies on
iterative recalling and reasoning of history to generate high-quality
responses. However, such repeated recall-reason steps easily produce biased
thoughts, \textit{i.e.}, inconsistent reasoning results when recalling the same
history for different questions. On the contrary, humans can keep thoughts in
the memory and recall them without repeated reasoning. Motivated by this human
capability, we propose a novel memory mechanism called TiM (Think-in-Memory)
that enables LLMs to maintain an evolved memory for storing historical thoughts
along the conversation stream. The TiM framework consists of two crucial
stages: (1) before generating a response, a LLM agent recalls relevant thoughts
from memory, and (2) after generating a response, the LLM agent post-thinks and
incorporates both historical and new thoughts to update the memory. Thus, TiM
can eliminate the issue of repeated reasoning by saving the post-thinking
thoughts as the history. Besides, we formulate the basic principles to organize
the thoughts in memory based on the well-established operations,
(\textit{i.e.}, insert, forget, and merge operations), allowing for dynamic
updates and evolution of the thoughts. Furthermore, we introduce
Locality-Sensitive Hashing into TiM to achieve efficient retrieval for the
long-term conversations. We conduct qualitative and quantitative experiments on
real-world and simulated dialogues covering a wide range of topics,
demonstrating that equipping existing LLMs with TiM significantly enhances
their performance in generating responses for long-term interactions.

提出了一种称为 TiM（Think-in-Memory）的新型记忆机制，通过保存后思考的思想作为历史记录，使得 LLMs 能够维护进化的记忆以存储历史思想，并且通过引入局部敏感哈希实现了对长期对话的高效检索，从而显著提升了现有 LLMs 在生成长期互动响应方面的性能。