Transformers have demonstrated their success in various domains and tasks.
However, Transformers struggle with long input sequences due to their limited
capacity. While one solution is to increase input length, endlessly stretching
the length is unrealistic. Furthermore, humans selectively remember and use
only relevant information from inputs, unlike Transformers which process all
raw data from start to end. We introduce Memoria, a general memory network that
applies Hebbian theory which is a major theory explaining human memory
formulation to enhance long-term dependencies in neural networks. Memoria
stores and retrieves information called engram at multiple memory levels of
working memory, short-term memory, and long-term memory, using connection
weights that change according to Hebb's rule. Through experiments with popular
Transformer-based models like BERT and GPT, we present that Memoria
significantly improves the ability to consider long-term dependencies in
various tasks. Results show that Memoria outperformed existing methodologies in
sorting and language modeling, and long text classification.

通过引入奇异值理论 (memory theory) 以增强神经网络的长期相关性，Memoria 在处理长文本和记忆力相关任务上显著优于传统的 Transformer 模型。

Memoria: 人类类似序列处理的 Hebbian 记忆架构

Memoria: Hebbian Memory Architecture for Human-Like Sequential  Processing

Hebbian theory seeks to explain how the neurons in the brain adapt to
stimuli, to enable learning. An interesting feature of Hebbian learning is that
it is an unsupervised method and as such, does not require feedback, making it
suitable in contexts where systems have to learn autonomously. This paper
explores how molecular systems can be designed to show such proto-intelligent
behaviours, and proposes the first chemical reaction network (CRN) that can
exhibit autonomous Hebbian learning across arbitrarily many input channels. The
system emulates a spiking neuron, and we demonstrate that it can learn
statistical biases of incoming inputs. The basic CRN is a minimal,
thermodynamically plausible set of micro-reversible chemical equations that can
be analysed with respect to their energy requirements. However, to explore how
such chemical systems might be engineered de novo, we also propose an extended
version based on enzyme-driven compartmentalised reactions. Finally, we also
show how a purely DNA system, built upon the paradigm of DNA strand
displacement, can realise neuronal dynamics. Our analysis provides a compelling
blueprint for exploring autonomous learning in biological settings, bringing us
closer to realising real synthetic biological intelligence.

论文探讨了如何设计分子系统以展现类似于自主学习的行为，提出了一种可对任意多个输入通道展现自主型 Hebb 学习的化学反应网络，并指出在构建新的化学系统和 DNA 系统中如何实现神经元动力学。

编程分子系统以模拟学习尖峰神经元

Programming molecular systems to emulate a learning spiking neuron

Dynamic model pruning is a recent direction that allows for the inference of
a different sub-network for each input sample during deployment. However,
current dynamic methods rely on learning a continuous channel gating through
regularization by inducing sparsity loss. This formulation introduces
complexity in balancing different losses (e.g task loss, regularization loss).
In addition, regularization based methods lack transparent tradeoff
hyperparameter selection to realize a computational budget. Our contribution is
two-fold: 1) decoupled task and pruning losses. 2) Simple hyperparameter
selection that enables FLOPs reduction estimation before training. Inspired by
the Hebbian theory in Neuroscience: "neurons that fire together wire together",
we propose to predict a mask to process k filters in a layer based on the
activation of its previous layer. We pose the problem as a self-supervised
binary classification problem. Each mask predictor module is trained to predict
if the log-likelihood for each filter in the current layer belongs to the top-k
activated filters. The value k is dynamically estimated for each input based on
a novel criterion using the mass of heatmaps. We show experiments on several
neural architectures, such as VGG, ResNet and MobileNet on CIFAR and ImageNet
datasets. On CIFAR, we reach similar accuracy to SOTA methods with 15% and 24%
higher FLOPs reduction. Similarly in ImageNet, we achieve lower drop in
accuracy with up to 13% improvement in FLOPs reduction.

本研究提出了一种基于 Hebbian 理论的动态网络剪枝方法，通过训练一个自我监督的二分类模型来预测每个层中需要处理的卷积核，从而实现 FLOPs 的减少，同时避免通过规范化引入复杂度来平衡各种损失，并实现透明的超参数选择。该方法在 CIFAR 和 ImageNet 数据集上的实验结果表明，与 SOTA 方法相比，具有相似的准确度和更高的 FLOPs 减少率。