Large language models (LLMs) have advanced in large strides due to the
effectiveness of the self-attention mechanism that processes and compares all
tokens at once. However, this mechanism comes with a fundamental issue -- the
predetermined context window is bound to be limited. Despite attempts to extend
the context window through methods like extrapolating the positional embedding,
using recurrence, or selectively retrieving essential parts of the long
sequence, long-text understanding continues to be a challenge. We propose an
alternative approach which instead treats the LLM as an interactive agent,
allowing it to decide how to read the text via iterative prompting. We
introduce MemWalker, a method that first processes the long context into a tree
of summary nodes. Upon receiving a query, the model navigates this tree in
search of relevant information, and responds once it gathers sufficient
information. On long-text question answering tasks our method outperforms
baseline approaches that use long context windows, recurrence, and retrieval.
We show that, beyond effective reading, MemWalker enhances explainability by
highlighting the reasoning steps as it interactively reads the text;
pinpointing the relevant text segments related to the query.

我们提出了 MemWalker，这是一种将长上下文处理成摘要节点树的方法，模型通过迭代提示的方式导航该树以寻找相关信息并一旦收集足够信息便回答问题，在长文本问答任务上，我们方法的性能优于使用长上下文窗口、重复和检索的基线方法。通过交互式阅读文本，MemWalker 还提升了解释能力，突出了推理步骤，并准确指出与查询相关的文本片段。