Recent NLP studies reveal that substantial linguistic information can be attributed to single neurons, i.e., individual dimensions of the representation vectors. We hypothesize that modeling strong interactions among neurons helps to better capture complex information by composing the linguistic properties embedded in individual neurons. Starting from this intuition, we propose a novel approach to compose representations learned by different components in neural machine translation (e.g., multi-layer networks or multi-head attention), based on modeling strong interactions among neurons in the representation vectors. Specifically, we leverage bilinear pooling to model pairwise multiplicative interactions among individual neurons, and a low-rank approximation to make the model computationally feasible. We further propose extended bilinear pooling to incorporate first-order representations. Experiments on WMT14 English-German and English-French translation tasks show that our model consistently improves performances over the SOTA Transformer baseline. Further analyses demonstrate that our approach indeed captures more syntactic and semantic information as expected.

通过模拟单个神经元之间的强交互作用，提出了一种新的方法来组成神经机器翻译中不同组件所学习的表示，该方法利用了双线性汇聚来建模单个神经元之间的成对乘法交互作用，并使用低秩近似使模型具有可计算性，并且在WMT14英德和英法翻译任务上得到了更好的实验结果。

基于神经元交互的表示合成的神经机器翻译