The legality of training language models (LMs) on copyrighted or otherwise
restricted data is under intense debate. However, as we show, model performance
significantly degrades if trained only on low-risk text (e.g., out-of-copyright
books or government documents), due to its limited size and domain coverage. We
present SILO, a new language model that manages this risk-performance tradeoff
during inference. SILO is built by (1) training a parametric LM on Open License
Corpus (OLC), a new corpus we curate with 228B tokens of public domain and
permissively licensed text and (2) augmenting it with a more general and easily
modifiable nonparametric datastore (e.g., containing copyrighted books or news)
that is only queried during inference. The datastore allows use of high-risk
data without training on it, supports sentence-level data attribution, and
enables data producers to opt out from the model by removing content from the
store. These capabilities can foster compliance with data-use regulations such
as the fair use doctrine in the United States and the GDPR in the European
Union. Our experiments show that the parametric LM struggles on domains not
covered by OLC. However, access to the datastore greatly improves out of domain
performance, closing 90% of the performance gap with an LM trained on the Pile,
a more diverse corpus with mostly high-risk text. We also analyze which
nonparametric approach works best, where the remaining errors lie, and how
performance scales with datastore size. Our results suggest that it is possible
to build high quality language models while mitigating their legal risk.

通过在开放许可的文本语料库（OLC）上训练参数化的语言模型，并在推断过程中使用包含有版权内容的数据存储库，SILO 模型能够在满足数据使用法规（如美国的公平使用原则和欧盟的 GDPR）的同时提高模型的性能。

SILO 语言模型：在非参数化数据存储中隔离法律风险

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore

The importance of learning rate (LR) schedules on network pruning has been
observed in a few recent works. As an example, Frankle and Carbin (2019)
highlighted that winning tickets (i.e., accuracy preserving subnetworks) can
not be found without applying a LR warmup schedule and Renda, Frankle and
Carbin (2020) demonstrated that rewinding the LR to its initial state at the
end of each pruning cycle improves performance. In this paper, we go one step
further by first providing a theoretical justification for the surprising
effect of LR schedules. Next, we propose a LR schedule for network pruning
called SILO, which stands for S-shaped Improved Learning rate Optimization. The
advantages of SILO over existing state-of-the-art (SOTA) LR schedules are
two-fold: (i) SILO has a strong theoretical motivation and dynamically adjusts
the LR during pruning to improve generalization. Specifically, SILO increases
the LR upper bound (max_lr) in an S-shape. This leads to an improvement of 2% -
4% in extensive experiments with various types of networks (e.g., Vision
Transformers, ResNet) on popular datasets such as ImageNet, CIFAR-10/100. (ii)
In addition to the strong theoretical motivation, SILO is empirically optimal
in the sense of matching an Oracle, which exhaustively searches for the optimal
value of max_lr via grid search. We find that SILO is able to precisely adjust
the value of max_lr to be within the Oracle optimized interval, resulting in
performance competitive with the Oracle with significantly lower complexity.

本文提出了一种名为 SILO 的改进型学习率优化策略来优化网络剪枝，通过在剪枝理论上给出理论证明，并在各种网络和流行数据集（例如 ImageNet，CIFAR-10 / 100）上进行广泛实验，获得了 2％-4％的性能提升。