In this work, we present a general framework for continual learning of
sequentially arrived tasks with the use of pre-training, which has emerged as a
promising direction for artificial intelligence systems to accommodate
real-world dynamics. From a theoretical perspective, we decompose its objective
into three hierarchical components, including within-task prediction,
task-identity inference, and task-adaptive prediction. Then we propose an
innovative approach to explicitly optimize these components with
parameter-efficient fine-tuning (PEFT) techniques and representation
statistics. We empirically demonstrate the superiority and generality of our
approach in downstream continual learning, and further explore the
applicability of PEFT techniques in upstream continual learning. We also
discuss the biological basis of the proposed framework with recent advances in
neuroscience.

通过使用预训练的方法，我们提出了一个通用的框架来持续学习连续到达的任务，从一个理论的角度，我们将其目标分解为三个层次的组成部分，包括任务内预测、任务标识推断和任务适应预测，并提出了一种创新的方法来显式优化这些组件，通过参数高效的微调技术和表示统计量，我们在下游连续学习中实证展示了我们方法的优越性和普遍性，并进一步探讨了在上游连续学习中应用参数高效的微调技术的可行性，同时结合神经科学中的最新进展，讨论了所提框架的生物学基础。