Neural models have yielded state-of-the-art results in deciphering spoken
language understanding (SLU) problems; however, these models require a
significant amount of domain-specific labeled examples for training, which is
prohibitively expensive. While pre-trained language models like BERT have been
shown to capture a massive amount of knowledge by learning from unlabeled
corpora and solve SLU using fewer labeled examples for adaption, the encoding
of knowledge is implicit and agnostic to downstream tasks. Such encoding
results in model inefficiencies in parameter usage: an entirely new model is
required for every domain. To address these challenges, we introduce a novel
SLU framework, comprising a conversational language modeling (CLM) pre-training
task and a light encoder architecture. The CLM pre-training enables networks to
capture the representation of the language in conversation style with the
presence of ASR errors. The light encoder architecture separates the shared
pre-trained networks from the mappings of generally encoded knowledge to
specific domains of SLU, allowing for the domain adaptation to be performed
solely at the light encoder and thus increasing efficiency. With the framework,
we match the performance of state-of-the-art SLU results on Alexa internal
datasets and on two public ones (ATIS, SNIPS), adding only 4.4% parameters per
task.

为解决领域特定标注样本数量过多的问题，提出了一种由对话语言建模预训练任务和轻量级编码器构成的 SLU 框架，通过预训练学习对话语言表示和解耦共享网络与映射领域特定知识的轻量级编码器实现领域适应。在内部和公共数据集上，该框架在添加少量参数的情况下与现有 SLU 方法的表现相匹配。