Compositionality is an important feature of discrete symbolic systems, such
as language and programs, as it enables them to have infinite capacity despite
a finite symbol set. It serves as a useful abstraction for reasoning in both
cognitive science and in AI, yet the interface between continuous and symbolic
processing is often imposed by fiat at the algorithmic level, such as by means
of quantization or a softmax sampling step. In this work, we explore how
discretization could be implemented in a more neurally plausible manner through
the modeling of attractor dynamics that partition the continuous representation
space into basins that correspond to sequences of symbols. Building on
established work in attractor networks and introducing novel training methods,
we show that imposing structure in the symbolic space can produce
compositionality in the attractor-supported representation space of rich
sensory inputs. Lastly, we argue that our model exhibits the process of an
information bottleneck that is thought to play a role in conscious experience,
decomposing the rich information of a sensory input into stable components
encoding symbolic information.

通过吸引子动力学对连续表示空间进行分区，并在符号空间中实施结构，可以实现丰富感官输入的吸引子支持表示空间中的组合性。

通过吸引子动力学构建离散、组合和符号化表征

Discrete, compositional, and symbolic representations through attractor  dynamics

A central challenge faced by memory systems is the robust retrieval of a
stored pattern in the presence of interference due to other stored patterns and
noise. A theoretically well-founded solution to robust retrieval is given by
attractor dynamics, which iteratively clean up patterns during recall. However,
incorporating attractor dynamics into modern deep learning systems poses
difficulties: attractor basins are characterised by vanishing gradients, which
are known to make training neural networks difficult. In this work, we avoid
the vanishing gradient problem by training a generative distributed memory
without simulating the attractor dynamics. Based on the idea of memory writing
as inference, as proposed in the Kanerva Machine, we show that a
likelihood-based Lyapunov function emerges from maximising the variational
lower-bound of a generative memory. Experiments shows it converges to correct
patterns upon iterative retrieval and achieves competitive performance as both
a memory model and a generative model.

本研究展示了一个基于 Kanerva 机器提出的记忆写入即推理的思想，通过训练一个生成式分布式记忆，最大化其变分下界导出一个儒雅彼诺夫函数。实验结果表明，在迭代读取中收敛到正确的模式，并且在记忆模型和生成模型方面都具有有竞争力的性能。