Due to the rapid generation and dissemination of information, large language
models (LLMs) quickly run out of date despite enormous development costs. Due
to this crucial need to keep models updated, online learning has emerged as a
critical necessity when utilizing LLMs for real-world applications. However,
given the ever-expanding corpus of unseen documents and the large parameter
space of modern LLMs, efficient adaptation is essential. To address these
challenges, we propose Memory of Amortized Contexts (MAC), an efficient and
effective online adaptation framework for LLMs with strong knowledge retention.
We propose an amortized feature extraction and memory-augmentation approach to
compress and extract information from new documents into compact modulations
stored in a memory bank. When answering questions, our model attends to and
extracts relevant knowledge from this memory bank. To learn informative
modulations in an efficient manner, we utilize amortization-based
meta-learning, which substitutes the optimization process with a single forward
pass of the encoder. Subsequently, we learn to choose from and aggregate
selected documents into a single modulation by conditioning on the question,
allowing us to adapt a frozen language model during test time without requiring
further gradient updates. Our experiment demonstrates the superiority of MAC in
multiple aspects, including online adaptation performance, time, and memory
efficiency. Code is available at: this https URL

大型语言模型的在线适应性方案，通过压缩与提取新文档中的信息，并存储在记忆库中，实现知识保留、问答和适应性，以提高效率和性能。