This paper introduces a new biologically-inspired training method named
Continual Learning through Adjustment Suppression and Sparsity Promotion
(CLASSP). CLASSP is based on two main principles observed in neuroscience,
particularly in the context of synaptic transmission and Long-Term Potentiation
(LTP). The first principle is a decay rate over the weight adjustment, which is
implemented as a generalization of the AdaGrad optimization algorithm. This
means that weights that have received many updates should have lower learning
rates as they likely encode important information about previously seen data.
However, this principle results in a diffuse distribution of updates throughout
the model, as it promotes updates for weights that haven't been previously
updated, while a sparse update distribution is preferred to leave weights
unassigned for future tasks. Therefore, the second principle introduces a
threshold on the loss gradient. This promotes sparse learning by updating a
weight only if the loss gradient with respect to that weight is above a certain
threshold, i.e. only updating weights with a significant impact on the current
loss. Both principles reflect phenomena observed in LTP, where a threshold
effect and a gradual saturation of potentiation have been observed. CLASSP is
implemented in a Python/PyTorch class, making it applicable to any model. When
compared with Elastic Weight Consolidation (EWC) using Computer Vision
datasets, CLASSP demonstrates superior performance in terms of accuracy and
memory footprint.

本文介绍了一种新的生物启发式训练方法，名为连续学习通过调整抑制和稀疏促进（CLASSP）。CLASSP 基于神经科学中观察到的两个主要原则，尤其是在突触传递和长期增强（LTP）的背景下。通过与计算机视觉数据集上的弹性权重合并（EWC）进行比较，CLASSP 在准确性和内存占用方面表现出更好的性能。

CLASSP：一种通过调整抑制和稀疏推动进行持续学习的生物启发方法

CLASSP: a Biologically-Inspired Approach to Continual Learning through  Adjustment Suppression and Sparsity Promotion

We study a class of non-convex and non-smooth problems with \textit{rank}
regularization to promote sparsity in optimal solution. We propose to apply the
proximal gradient descent method to solve the problem and accelerate the
process with a novel support set projection operation on the singular values of
the intermediate update. We show that our algorithms are able to achieve a
convergence rate of $O(\frac{1}{t^2})$, which is exactly same as Nesterov's
optimal convergence rate for first-order methods on smooth and convex problems.
Strict sparsity can be expected and the support set of singular values during
each update is monotonically shrinking, which to our best knowledge, is novel
in momentum-based algorithms.

我们研究了一类具有秩规范化的非凸非光滑问题，以促进最优解的稀疏性。我们提出了应用近端梯度下降方法来解决该问题，并通过一种新型的支持集投影操作加速过程，作用于中间更新的奇异值。我们展示了我们的算法能够达到收敛速度为 $O (rac {1}{t^2})$，这与 Nesterov 有关于光滑凸问题的一阶方法的最优收敛速度完全相同。可以期望严格的稀疏性，且在每次更新中奇异值的支持集是单调收缩的，这在基于动量的算法中是新颖的。

严格低秩约束优化 -- 一种渐进 $O (rac {1}{t^2})$ 方法

Strictly Low Rank Constraint Optimization -- An Asymptotically  $\mathcal{O}(\frac{1}{t^2})$ Method

The iterations of many first-order algorithms, when applied to minimizing
common regularized regression functions, often resemble neural network layers
with pre-specified weights. This observation has prompted the development of
learning-based approaches that purport to replace these iterations with
enhanced surrogates forged as DNN models from available training data. For
example, important NP-hard sparse estimation problems have recently benefitted
from this genre of upgrade, with simple feedforward or recurrent networks
ousting proximal gradient-based iterations. Analogously, this paper
demonstrates that more powerful Bayesian algorithms for promoting sparsity,
which rely on complex multi-loop majorization-minimization techniques, mirror
the structure of more sophisticated long short-term memory (LSTM) networks, or
alternative gated feedback networks previously designed for sequence
prediction. As part of this development, we examine the parallels between
latent variable trajectories operating across multiple time-scales during
optimization, and the activations within deep network structures designed to
adaptively model such characteristic sequences. The resulting insights lead to
a novel sparse estimation system that, when granted training data, can estimate
optimal solutions efficiently in regimes where other algorithms fail, including
practical direction-of-arrival (DOA) and 3D geometry recovery problems. The
underlying principles we expose are also suggestive of a learning process for a
richer class of multi-loop algorithms in other domains.

本文指出，关于促进稀疏性的更强大的贝叶斯算法具有类似于长短期记忆 (LSTM) 网络或先前设计用于序列预测的替代门控反馈网络的结构，从而导致了一种新的稀疏估计系统，当授予训练数据时，可以在其他算法失败的方案中高效地估计最优解，包括在实际方向 - 到达 (DOA) 和三维几何恢复问题中。