Compositionality is an important feature of discrete symbolic systems, such
as language and programs, as it enables them to have infinite capacity despite
a finite symbol set. It serves as a useful abstraction for reasoning in both
cognitive science and in AI, yet the interface between continuous and symbolic
processing is often imposed by fiat at the algorithmic level, such as by means
of quantization or a softmax sampling step. In this work, we explore how
discretization could be implemented in a more neurally plausible manner through
the modeling of attractor dynamics that partition the continuous representation
space into basins that correspond to sequences of symbols. Building on
established work in attractor networks and introducing novel training methods,
we show that imposing structure in the symbolic space can produce
compositionality in the attractor-supported representation space of rich
sensory inputs. Lastly, we argue that our model exhibits the process of an
information bottleneck that is thought to play a role in conscious experience,
decomposing the rich information of a sensory input into stable components
encoding symbolic information.

通过吸引子动力学对连续表示空间进行分区，并在符号空间中实施结构，可以实现丰富感官输入的吸引子支持表示空间中的组合性。

通过吸引子动力学构建离散、组合和符号化表征

Discrete, compositional, and symbolic representations through attractor  dynamics

The wealth of structured (e.g. Wikidata) and unstructured data about the
world available today presents an incredible opportunity for tomorrow's
Artificial Intelligence. So far, integration of these two different modalities
is a difficult process, involving many decisions concerning how best to
represent the information so that it will be captured or useful, and
hand-labeling large amounts of data. DeepType overcomes this challenge by
explicitly integrating symbolic information into the reasoning process of a
neural network with a type system. First we construct a type system, and
second, we use it to constrain the outputs of a neural network to respect the
symbolic structure. We achieve this by reformulating the design problem into a
mixed integer problem: create a type system and subsequently train a neural
network with it. In this reformulation discrete variables select which
parent-child relations from an ontology are types within the type system, while
continuous variables control a classifier fit to the type system. The original
problem cannot be solved exactly, so we propose a 2-step algorithm: 1)
heuristic search or stochastic optimization over discrete variables that define
a type system informed by an Oracle and a Learnability heuristic, 2) gradient
descent to fit classifier parameters. We apply DeepType to the problem of
Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC
KBP 2010) and find that it outperforms all existing solutions by a wide margin,
including approaches that rely on a human-designed type system or recent deep
learning-based entity embeddings, while explicitly using symbolic information
lets it integrate new entities without retraining.

DeepType 是一种将符号信息显式地集成到神经网络推理过程中的模型，它通过构建类型系统并将其用于限制神经网络输出的符号结构来解决判断结构以及有效地捕捉信息的问题。在实验中，它用于 Entity Linking 任务，优于人工设计类型系统或最近使用深度学习相似性度量的方案，同时使用符号信息可以无需重新训练即可集成新实体。

DeepType: 神经类型系统发展的多语言实体链接

DeepType: Multilingual Entity Linking by Neural Type System Evolution

In order for neural networks to learn complex languages or grammars, they
must have sufficient computational power or resources to recognize or generate
such languages. Though many approaches have been discussed, one ob- vious
approach to enhancing the processing power of a recurrent neural network is to
couple it with an external stack memory - in effect creating a neural network
pushdown automata (NNPDA). This paper discusses in detail this NNPDA - its
construction, how it can be trained and how useful symbolic information can be
extracted from the trained network.
In order to couple the external stack to the neural network, an optimization
method is developed which uses an error function that connects the learning of
the state automaton of the neural network to the learning of the operation of
the external stack. To minimize the error function using gradient descent
learning, an analog stack is designed such that the action and storage of
information in the stack are continuous. One interpretation of a continuous
stack is the probabilistic storage of and action on data. After training on
sample strings of an unknown source grammar, a quantization procedure extracts
from the analog stack and neural network a discrete pushdown automata (PDA).
Simulations show that in learning deterministic context-free grammars - the
balanced parenthesis language, 1*n0*n, and the deterministic Palindrome - the
extracted PDA is correct in the sense that it can correctly recognize unseen
strings of arbitrary length. In addition, the extracted PDAs can be shown to be
identical or equivalent to the PDAs of the source grammars which were used to
generate the training strings.

本研究讨论了一种利用外部栈内存扩展循环神经网络处理能力的方法，即神经网络下推自动机（NNPDA），并详细介绍了它的构建、训练和从训练网络中提取象征信息的能力。通过样本字符串的训练，可以提取出离散下推自动机（PDA），能够准确地识别未知来源语法的任意长度的字符串，并且与原始语法的 PDA 相同或等价。