Developing neural architectures that are capable of logical reasoning has
become increasingly important for a wide range of applications (e.g., natural
language processing). Towards this grand objective, we propose a symbolic
reasoning architecture that chains many join operators together to model output
logical expressions. In particular, we demonstrate that such an ensemble of
join-chains can express a broad subset of ''tree-structured'' first-order
logical expressions, named FOET, which is particularly useful for modeling
natural languages. To endow it with differentiable learning capability, we
closely examine various neural operators for approximating the symbolic
join-chains. Interestingly, we find that the widely used multi-head
self-attention module in transformer can be understood as a special neural
operator that implements the union bound of the join operator in probabilistic
predicate space. Our analysis not only provides a new perspective on the
mechanism of the pretrained models such as BERT for natural language
understanding but also suggests several important future improvement
directions.

提出一种符号推理架构，使用许多连接算子来模拟输出逻辑表达式，并演示这样的连接 - 链集合可以表达广泛子集的一阶逻辑表达式，名为 FOET，特别适用于建模自然语言。