Continual learning is a problem for artificial neural networks that their
biological counterparts are adept at solving. Building on work using Sparse
Distributed Memory (SDM) to connect a core neural circuit with the powerful
Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is
a strong continual learner. We find that every component of our MLP variant
translated from biology is necessary for continual learning. Our solution is
also free from any memory replay or task information, and introduces novel
methods to train sparse networks that may be broadly applicable.

使用稀疏分布式记忆连接核心神经回路与 Transformer 模型，创造了一种修改的多层感知器，该算法可有效解决人工神经网络所面临的连续学习问题，并广泛适用于训练稀疏网络。

稀疏分布式内存是一种连续学习器

Sparse Distributed Memory is a Continual Learner

While Attention has come to be an important mechanism in deep learning, there
remains limited intuition for why it works so well. Here, we show that
Transformer Attention can be closely related under certain data conditions to
Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative
memory model. We confirm that these conditions are satisfied in pre-trained
GPT2 Transformer models. We discuss the implications of the Attention-SDM map
and provide new computational and biological interpretations of Attention.

该研究发现，在某些数据条件下，Transformer Attention 机制与 Kanerva 的 Sparse Distributed Memory 有密切关联，可以提供 Attention 的新的计算和生物学解释，进一步确认预训练的 GPT2 Transformer 模型已满足这些条件。