Catastrophic forgetting remains a significant challenge to continual learning
for decades. While recent works have proposed effective methods to mitigate
this problem, they mainly focus on the algorithmic side. Meanwhile, we do not
fully understand what architectural properties of neural networks lead to
catastrophic forgetting. This study aims to fill this gap by studying the role
of activation functions in the training dynamics of neural networks and their
impact on catastrophic forgetting. Our study reveals that, besides sparse
representations, the gradient sparsity of activation functions also plays an
important role in reducing forgetting. Based on this insight, we propose a new
class of activation functions, elephant activation functions, that can generate
both sparse representations and sparse gradients. We show that by simply
replacing classical activation functions with elephant activation functions, we
can significantly improve the resilience of neural networks to catastrophic
forgetting. Our method has broad applicability and benefits for continual
learning in regression, class incremental learning, and reinforcement learning
tasks. Specifically, we achieves excellent performance on Split MNIST dataset
in just one single pass, without using replay buffer, task boundary
information, or pre-training.

通过研究神经网络的激活函数在训练动态和遗忘问题上的影响，本研究揭示了稀疏表示和激活函数的梯度稀疏性在减少遗忘方面的重要作用，并提出了一种名为 elephant 激活函数的新类别的激活函数，它能够生成稀疏表示和稀疏梯度，通过用 elephant 激活函数替换传统激活函数，可以显著提高神经网络对遗忘的韧性。该方法在回归、类递增学习和强化学习任务中都具有广泛的适用性和益处，在 Split MNIST 数据集上只需一个遍历就能取得出色的性能，而无需使用回放缓冲区、任务边界信息或预训练。