Intent-detection and slot-filling are the two main tasks in natural language
understanding. In this study, we propose CTRAN, a novel encoder-decoder
CNN-Transformer-based architecture for intent-detection and slot-filling. In
the encoder, we use BERT, followed by several convolutional layers, and
rearrange the output using window feature sequence. We use stacked Transformer
encoders after the window feature sequence. For the intent-detection decoder,
we utilize self-attention followed by a linear layer. In the slot-filling
decoder, we introduce the aligned Transformer decoder, which utilizes a zero
diagonal mask, aligning output tags with input tokens. We apply our network on
ATIS and SNIPS, and surpass the current state-of-the-art in slot-filling on
both datasets. Furthermore, we incorporate the language model as word
embeddings, and show that this strategy yields a better result when compared to
the language model as an encoder.

本研究提出了基于 CTAN（一种新型 encoder-decoder CNN-Transformer-based 结构）的自然语言理解方法，它在使用 BERT 编码器后、多个卷积层之后使用窗口特征序列重新排列输出，并在窗口特征序列之后使用堆叠的 Transformer 编码器。在意图检测解码器中，使用自我注意力和一个线性层，在填充槽解码器中，引入对齐 Transformer 解码器。结果表明我们的模型在 ATIS 和 SNIPS 中对填充槽任务超过了目前的最优。另外，研究还表明将语言模型作为单词嵌入优于将其作为编码器。