Task-oriented dialog systems need to know when a query falls outside their
range of supported intents, but current text classification corpora only define
label sets that cover every example. We introduce a new dataset that includes
queries that are out-of-scope---i.e., queries that do not fall into any of the
system's supported intents. This poses a new challenge because models cannot
assume that every query at inference time belongs to a system-supported intent
class. Our dataset also covers 150 intent classes over 10 domains, capturing
the breadth that a production task-oriented agent must handle. We evaluate a
range of benchmark classifiers on our dataset along with several different
out-of-scope identification schemes. We find that while the classifiers perform
well on in-scope intent classification, they struggle to identify out-of-scope
queries. Our dataset and evaluation fill an important gap in the field,
offering a way of more rigorously and realistically benchmarking text
classification in task-driven dialog systems.

该研究旨在通过引入包括超出支持意图范围的查询的新数据集来评估范围内 / 超出范围的分类器及其性能，以更严谨、更真实地评估用于任务驱动对话系统中的文本分类。