Rehearsal approaches enjoy immense popularity with Continual Learning (CL)
practitioners. These methods collect samples from previously encountered data
distributions in a small memory buffer; subsequently, they repeatedly optimize
on the latter to prevent catastrophic forgetting. This work draws attention to
a hidden pitfall of this widespread practice: repeated optimization on a small
pool of data inevitably leads to tight and unstable decision boundaries, which
are a major hindrance to generalization. To address this issue, we propose
Lipschitz-DrivEn Rehearsal (LiDER), a surrogate objective that induces
smoothness in the backbone network by constraining its layer-wise Lipschitz
constants w.r.t. replay examples. By means of extensive experiments, we show
that applying LiDER delivers a stable performance gain to several
state-of-the-art rehearsal CL methods across multiple datasets, both in the
presence and absence of pre-training. Through additional ablative experiments,
we highlight peculiar aspects of buffer overfitting in CL and better
characterize the effect produced by LiDER. Code is available at
this https URL

通过在重播示例中对网络的每个层进行 Lipschitz 限制，Lipschitz-DrivEn Rehearsal (LiDER) 提出了一个监督目标，以诱导平滑的反向传播，从而防止缓冲区过度拟合引起的紧张和不稳定的决策边界，并表明它能够提供稳定的绩效提升。

利普希茨驱动的回顾法在不断学习中的有效性

On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning

Despite significant advances, continual learning models still suffer from
catastrophic forgetting when exposed to incrementally available data from
non-stationary distributions. Rehearsal approaches alleviate the problem by
maintaining and replaying a small episodic memory of previous samples, often
implemented as an array of independent memory slots. In this work, we propose
to augment such an array with a learnable random graph that captures pairwise
similarities between its samples, and use it not only to learn new tasks but
also to guard against forgetting. Empirical results on several benchmark
datasets show that our model consistently outperforms recently proposed
baselines for task-free continual learning.

本文提出通过增加可学习的随机图来维护和重放以前样本的小型陈情表记忆，来解决非静态分布中可用数据的持续学习问题的方法。在多个基准数据集上的实证结果表明，该模型持续胜过最近提出的基线方法，用于无任务的持续学习。