Loss of plasticity is a phenomenon where neural networks become more
difficult to train during the course of learning. Continual learning algorithms
seek to mitigate this effect by sustaining good predictive performance while
maintaining network trainability. We develop new techniques for improving
continual learning by first reconsidering how initialization can ensure
trainability during early phases of learning. From this perspective, we derive
new regularization strategies for continual learning that ensure beneficial
initialization properties are better maintained throughout training. In
particular, we investigate two new regularization techniques for continual
learning: (i) Wasserstein regularization toward the initial weight
distribution, which is less restrictive than regularizing toward initial
weights; and (ii) regularizing weight matrix singular values, which directly
ensures gradient diversity is maintained throughout training. We present an
experimental analysis that shows these alternative regularizers can improve
continual learning performance across a range of supervised learning tasks and
model architectures. The alternative regularizers prove to be less sensitive to
hyperparameters while demonstrating better training in individual tasks,
sustaining trainability as new tasks arrive, and achieving better
generalization performance.

神经网络的塑性缺失意味着在学习过程中网络训练变得更加困难。我们通过改进初始化技术、提出新的正则化策略以及保持梯度多样性来改善持续学习，并证明这些替代技术在不同的监督学习任务和模型结构上能够提高持续学习性能。

通过谱正则化实现持续学习

Learning Continually by Spectral Regularization

This paper addresses the training of Neural Ordinary Differential Equations
(neural ODEs), and in particular explores the interplay between numerical
integration techniques, stability regions, step size, and initialization
techniques. It is shown how the choice of integration technique implicitly
regularizes the learned model, and how the solver's corresponding stability
region affects training and prediction performance. From this analysis, a
stability-informed parameter initialization technique is introduced. The
effectiveness of the initialization method is displayed across several learning
benchmarks and industrial applications.

本文研究了神经常微分方程（Neural Ordinary Differential Equations）的训练，并探讨了数值积分技术、稳定区域、步长和初始化技术之间的相互作用。文章展示了积分技术的选择如何隐式地对学习模型进行正则化，并且求解器的相应稳定区域如何影响训练和预测性能。在此分析的基础上，引入了一种基于稳定性的参数初始化技术。该初始化方法在多个学习基准和工业应用中显示出效果。

神经常微分方程的稳定感知初始化

Stability-Informed Initialization of Neural Ordinary Differential  Equations

An eco-system of agents each having their own policy with some, but limited,
generalizability has proven to be a reliable approach to increase
generalization across procedurally generated environments. In such an approach,
new agents are regularly added to the eco-system when encountering a new
environment that is outside of the scope of the eco-system. The speed of
adaptation and general effectiveness of the eco-system approach highly depends
on the initialization of new agents. In this paper we propose different
initialization techniques, inspired from Deep Neural Network initialization and
transfer learning, and study their impact.

本研究提出了受深度神经网络初始化和迁移学习启发的不同初始化技术，探讨其对生态系统方法的速度和效果的影响。