We propose a novel perspective of the attention mechanism by reinventing it
as a memory architecture for neural networks, namely Neural Attention Memory
(NAM). NAM is a memory structure that is both readable and writable via
differentiable linear algebra operations. We explore three use cases of NAM:
memory-augmented neural network (MANN), few-shot learning, and efficient
long-range attention. First, we design two NAM-based MANNs of Long Short-term
Memory (LSAM) and NAM Turing Machine (NAM-TM) that show better computational
powers in algorithmic zero-shot generalization tasks compared to other
baselines such as differentiable neural computer (DNC). Next, we apply NAM to
the N-way K-shot learning task and show that it is more effective at reducing
false positives compared to the baseline cosine classifier. Finally, we
implement an efficient Transformer with NAM and evaluate it with long-range
arena tasks to show that NAM can be an efficient and effective alternative for
scaled dot-product attention.

提出了一种新颖的关注机制，即神经关注记忆（NAM），它是一种可读写的记忆结构，并基于此设计了基于 NAM 的 MANN、few-shot 学习、N-way K-shot 学习和 Transformer 等模型，实验证明 NAM 在算法零样本泛化、减少假阳性等方面效果更好。