Forgetting is often seen as an unwanted characteristic in both human and
machine learning. However, we propose that forgetting can in fact be favorable
to learning. We introduce "forget-and-relearn" as a powerful paradigm for
shaping the learning trajectories of artificial neural networks. In this
process, the forgetting step selectively removes undesirable information from
the model, and the relearning step reinforces features that are consistently
useful under different conditions. The forget-and-relearn framework unifies
many existing iterative training algorithms in the image classification and
language emergence literature, and allows us to understand the success of these
algorithms in terms of the disproportionate forgetting of undesirable
information. We leverage this understanding to improve upon existing algorithms
by designing more targeted forgetting operations. Insights from our analysis
provide a coherent view on the dynamics of iterative training in neural
networks and offer a clear path towards performance improvements.

本文提出了遗忘与重新学习（forget-and-relearn）是一个有利于神经网络学习的新型方法，通过不断遗忘不必要的信息并强化对不同条件下有用的特征，来塑造模型的学习轨迹。其思想融合了图像分类和语言涌现领域的众多迭代训练算法，并通过遗忘不必要的信息来改善模型。通过分析，提出了关于神经网络迭代训练动态的一致性观点，并提供了一条清晰的路径向更高性能。