User interaction with voice-powered agents generates large amounts of
unlabeled utterances. In this paper, we explore techniques to efficiently
transfer the knowledge from these unlabeled utterances to improve model
performance on Spoken Language Understanding (SLU) tasks. We use Embeddings
from Language Model (ELMo) to take advantage of unlabeled data by learning
contextualized word representations. Additionally, we propose ELMo-Light
(ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our
findings suggest unsupervised pre-training on a large corpora of unlabeled
utterances leads to significantly better SLU performance compared to training
from scratch and it can even outperform conventional supervised transfer.
Additionally, we show that the gains from unsupervised transfer techniques can
be further improved by supervised transfer. The improvements are more
pronounced in low resource settings and when using only 1000 labeled in-domain
samples, our techniques match the performance of training from scratch on
10-15x more labeled in-domain data.

通过使用语言模型的嵌入（ELMo）技术，提出了一种更快、更简单的无监督预训练方法 ELMo-Light（ELMoL），在大量未标记的句子上进行的无监督预训练，在语音识别任务上比从头开始训练或传统的有监督迁移学习方法表现更好，并且当域内的标记样本仅有 1000 个时，我们的技术可以匹配使用 10-15 倍更多标记的样本从头开始训练的性能。