We explore active learning (AL) for improving the accuracy of new domains in
a natural language understanding (NLU) system. We propose an algorithm called
Majority-CRF that uses an ensemble of classification models to guide the
selection of relevant utterances, as well as a sequence labeling model to help
prioritize informative examples. Experiments with three domains show that
Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random
sampling with the same annotation budget, and statistically significant
improvements compared to other AL approaches. Additionally, case studies with
human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing
NLU system.

本文提出了一种名为 Majority-CRF 的算法，通过使用分类模型的集成来指导选取相关的话语，以及序列标注模型来帮助优先选择信息量大的示例，进行主动学习，实验结果表明，与随机采样相比，本算法在相同的注释预算下可以实现 6.6％-9％的相对误差率降低，并且与其他主动学习方法相比，取得了显著的改进，此外，在六个新领域进行人机交互的案例研究表明，本文提出的算法在现有的 NLU 系统上取得了 4.6％-9％的改进。