Large Language Models (LLMs) serve as repositories of extensive world
knowledge, enabling them to perform tasks such as question-answering and
fact-checking. However, this knowledge can become obsolete as global contexts
change. In this paper, we introduce a novel problem in the realm of continual
learning: Online Continual Knowledge Learning (OCKL). This problem formulation
aims to manage the dynamic nature of world knowledge in LMs under real-time
constraints. We propose a new benchmark and evaluation metric designed to
measure both the rate of new knowledge acquisition and the retention of
previously learned knowledge. Our empirical evaluation, conducted using a
variety of state-of-the-art methods, establishes robust base-lines for OCKL.
Our results reveal that existing continual learning approaches are
unfortunately insufficient for tackling the unique challenges posed by OCKL. We
identify key factors that influence the trade-off between knowledge acquisition
and retention, thereby advancing our understanding of how to train LMs in a
continually evolving environment.

在这篇论文中，我们介绍了一个新的问题，在持续学习领域中称为 “在线持续知识学习（OCKL）”。我们提出了一个新的基准和评估指标，旨在测量新知识获取的速度和先前学到的知识的保留率。通过使用各种最先进的方法进行的实证评估为 OCKL 建立了稳健的基准。我们的结果显示，现有的持续学习方法对于应对 OCKL 提出的独特挑战不足够。我们确定了影响知识获取和保留之间权衡的关键因素，从而推进了我们对如何在不断演化的环境中训练 LLMs 的理解。