Understanding the intricate operations of Recurrent Neural Networks (RNNs)
mechanistically is pivotal for advancing their capabilities and applications.
In this pursuit, we propose the Episodic Memory Theory (EMT), illustrating that
RNNs can be conceptualized as discrete-time analogs of the recently proposed
General Sequential Episodic Memory Model. To substantiate EMT, we introduce a
novel set of algorithmic tasks tailored to probe the variable binding behavior
in RNNs. Utilizing the EMT, we formulate a mathematically rigorous circuit that
facilitates variable binding in these tasks. Our empirical investigations
reveal that trained RNNs consistently converge to the variable binding circuit,
thus indicating universality in the dynamics of RNNs. Building on these
findings, we devise an algorithm to define a privileged basis, which reveals
hidden neurons instrumental in the temporal storage and composition of
variables, a mechanism vital for the successful generalization in these tasks.
We show that the privileged basis enhances the interpretability of the learned
parameters and hidden states of RNNs. Our work represents a step toward
demystifying the internal mechanisms of RNNs and, for computational
neuroscience, serves to bridge the gap between artificial neural networks and
neural memory models.

我们提出了 “情节记忆理论”，旨在揭示循环神经网络作为离散时间模拟的 “一般顺序情节记忆模型”，我们引入了一套适用于探索循环神经网络中变量绑定行为的新颖算法任务，并且通过数学上严格的电路形式加强了这些任务中的变量绑定。我们的实验研究表明，经过训练的循环神经网络始终收敛到变量绑定电路，从而揭示了循环神经网络动力学的普遍性，进而设计了一种算法来定义特权基础，这揭示了在时间存储和组合变量中起关键作用的隐藏神经元，这一机制对于成功的泛化至关重要。我们展示了特权基础如何提高了循环神经网络的学习参数和隐藏状态的可解释性。我们的工作在循环神经网络的内部机制解谜方面迈出了一步，并且对计算神经科学有助于弥合人工神经网络和神经记忆模型之间的差距。

递归神经网络的机制性解释的记忆片段理论

Episodic Memory Theory for the Mechanistic Interpretation of Recurrent  Neural Networks

Symbolic reasoning and neural networks are often considered incompatible
approaches. Connectionist models known as Vector Symbolic Architectures (VSAs)
can potentially bridge this gap. However, classical VSAs and neural networks
are still considered incompatible. VSAs encode symbols by dense pseudo-random
vectors, where information is distributed throughout the entire neuron
population. Neural networks encode features locally, often forming sparse
vectors of neural activation. Following Rachkovskij (2001); Laiho et al.
(2015), we explore symbolic reasoning with sparse distributed representations.
The core operations in VSAs are dyadic operations between vectors to express
variable binding and the representation of sets. Thus, algebraic manipulations
enable VSAs to represent and process data structures in a vector space of fixed
dimensionality. Using techniques from compressed sensing, we first show that
variable binding between dense vectors in VSAs is mathematically equivalent to
tensor product binding between sparse vectors, an operation which increases
dimensionality. This result implies that dimensionality-preserving binding for
general sparse vectors must include a reduction of the tensor matrix into a
single sparse vector. Two options for sparsity-preserving variable binding are
investigated. One binding method for general sparse vectors extends earlier
proposals to reduce the tensor product into a vector, such as circular
convolution. The other method is only defined for sparse block-codes,
block-wise circular convolution. Our experiments reveal that variable binding
for block-codes has ideal properties, whereas binding for general sparse
vectors also works, but is lossy, similar to previous proposals. We demonstrate
a VSA with sparse block-codes in example applications, cognitive reasoning and
classification, and discuss its relevance for neuroscience and neural networks.

本文探讨了 Vector Symbolic Architectures 中稀疏分布式表示下的符号推导，通过实验证明了基于块编码进行变量绑定的方法具有良好的性质，并在认知推理和分类等实际应用中展示了它的价值。