Large language models (LLMs) are routinely pre-trained on billions of tokens,
only to start the process over again once new data becomes available. A much
more efficient solution is to continually pre-train these models, saving
significant compute compared to re-training. However, the distribution shift
induced by new data typically results in degraded performance on previous data
or poor adaptation to the new data. In this work, we show that a simple and
scalable combination of learning rate (LR) re-warming, LR re-decaying, and
replay of previous data is sufficient to match the performance of fully
re-training from scratch on all available data, as measured by final loss and
language model (LM) evaluation benchmarks. Specifically, we show this for a
weak but realistic distribution shift between two commonly used LLM
pre-training datasets (English$\rightarrow$English) and a stronger distribution
shift (English$\rightarrow$German) at the $405$M parameter model scale with
large dataset sizes (hundreds of billions of tokens). Selecting the weak but
realistic shift for larger-scale experiments, we also find that our continual
learning strategies match the re-training baseline for a 10B parameter LLM. Our
results demonstrate that LLMs can be successfully updated via simple and
scalable continual learning strategies, matching the re-training baseline using
only a fraction of the compute. Finally, inspired by previous work, we propose
alternatives to the cosine learning rate schedule that help circumvent
forgetting induced by LR re-warming and that are not bound to a fixed token
budget.

大型语言模型的持续学习策略可以通过简单且可扩展的方法成功更新模型，同时只需使用一小部分计算资源来达到重新训练的效果。

简单且可扩展的策略，用于持续预训练大型语言模型

Simple and Scalable Strategies to Continually Pre-train Large Language  Models

Tiny machine learning (TinyML) in IoT systems exploits MCUs as edge devices
for data processing. However, traditional TinyML methods can only perform
inference, limited to static environments or classes. Real case scenarios
usually work in dynamic environments, thus drifting the context where the
original neural model is no more suitable. For this reason, pre-trained models
reduce accuracy and reliability during their lifetime because the data recorded
slowly becomes obsolete or new patterns appear. Continual learning strategies
maintain the model up to date, with runtime fine-tuning of the parameters. This
paper compares four state-of-the-art algorithms in two real applications: i)
gesture recognition based on accelerometer data and ii) image classification.
Our results confirm these systems' reliability and the feasibility of deploying
them in tiny-memory MCUs, with a drop in the accuracy of a few percentage
points with respect to the original models for unconstrained computing
platforms.

本文比较了四种最先进的算法在两个实际应用中的效果，其中之一是基于加速度计数据的手势识别，另一个是图像分类，结果证实了这些系统的可靠性和在微型存储器 MCUs 中部署的可行性。