Systematic compositionality is an essential mechanism in human language,
allowing the recombination of known parts to create novel expressions. However,
existing neural models have been shown to lack this basic ability in learning
symbolic structures. Motivated by the failure of a Transformer model on the
SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing
a command into actions, we propose two auxiliary sequence prediction tasks that
track the progress of function and argument semantics, as additional training
supervision. These automatically-generated sequences are more representative of
the underlying compositional symbolic structures of the input data. During
inference, the model jointly predicts the next action and the next tokens in
the auxiliary sequences at each step. Experiments on the SCAN dataset show that
our method encourages the Transformer to understand compositional structures of
the command, improving its accuracy on multiple challenging splits from <= 10%
to 100%. With only 418 (5%) training instances, our approach still achieves
97.8% accuracy on the MCD1 split. Therefore, we argue that compositionality can
be induced in Transformers given minimal but proper guidance. We also show that
a better result is achieved using less contextualized vectors as the
attention's query, providing insights into architecture choices in achieving
systematic compositionality. Finally, we show positive generalization results
on the groundedSCAN task (Ruis et al., 2020). Our code is publicly available
at: this https URL

该论文提出了两种辅助序列预测任务来帮助解决神经模型在学习符号结构时的不足，并通过在 SCAN 数据集上的实验证明了所提出的方法可以提高 Transformer 模型理解命令的组合结构的能力。