Pre-trained Transformer-based models were reported to be robust in intent
classification. In this work, we first point out the importance of in-domain
out-of-scope detection in few-shot intent recognition tasks and then illustrate
the vulnerability of pre-trained Transformer-based models against samples that
are in-domain but out-of-scope (ID-OOS). We construct two new datasets, and
empirically show that pre-trained models do not perform well on both ID-OOS
examples and general out-of-scope examples, especially on fine-grained few-shot
intent detection tasks. To figure out how the models mistakenly classify ID-OOS
intents as in-scope intents, we further conduct analysis on confidence scores
and the overlapping keywords, as well as point out several prospective
directions for future work. Resources are available on
this https URL

本文探讨了在少样本意图识别任务中，在领域内外范围检测的重要性，揭示了预训练 Transformer 模型在领域内但扩展到领域外数据上的漏洞，构建了新的数据集并对模型能力作出实证验证。同时，对模型错误分类的 ID-OOS 意图进行了分析，并提出了若干未来工作方向。