We propose a novel Transformer encoder-based architecture with syntactical
knowledge encoded for intent detection and slot filling. Specifically, we
encode syntactic knowledge into the Transformer encoder by jointly training it
to predict syntactic parse ancestors and part-of-speech of each token via
multi-task learning. Our model is based on self-attention and feed-forward
layers and does not require external syntactic information to be available at
inference time. Experiments show that on two benchmark datasets, our models
with only two Transformer encoder layers achieve state-of-the-art results.
Compared to the previously best performed model without pre-training, our
models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85%
for slot filling and intent detection on the SNIPS dataset, respectively. Our
models also achieve absolute F1 score and accuracy improvement of 0.1% and
0.34% for slot filling and intent detection on the ATIS dataset, respectively,
over the previously best performed model. Furthermore, the visualization of the
self-attention weights illustrates the benefits of incorporating syntactic
information during training.

本文提出了一种基于 Transformer 编码器的架构，用于意图检测和槽填充，并将句法知识编码到 Transformer 编码器中，通过多任务学习联合训练来预测每个标记的句法分析祖先和词性标注。实验结果表明，与之前最好的模型相比，我们的模型在两个基准数据集上实现了最先进的结果