Intent classifiers must be able to distinguish when a user's utterance does
not belong to any supported intent to avoid producing incorrect and unrelated
system responses. Although out-of-scope (OOS) detection for intent classifiers
has been studied, previous work has not yet studied changes in classifier
performance against hard-negative out-of-scope utterances (i.e., inputs that
share common features with in-scope data, but are actually out-of-scope). We
present an automated technique to generate hard-negative OOS data using
ChatGPT. We use our technique to build five new hard-negative OOS datasets, and
evaluate each against three benchmark intent classifiers. We show that
classifiers struggle to correctly identify hard-negative OOS utterances more
than general OOS utterances. Finally, we show that incorporating hard-negative
OOS data for training improves model robustness when detecting hard-negative
OOS data and general OOS data. Our technique, datasets, and evaluation address
an important void in the field, offering a straightforward and inexpensive way
to collect hard-negative OOS data and improve intent classifiers' robustness.

通过使用 ChatGPT 生成困难负向超出范围数据并使用之评估，该论文研究了意图分类器，特别是在识别困难负向超出范围情况时的性能变化以及在训练中加入困难负向超出范围数据对于提高模型鲁棒性的影响。

使用 ChatGPT 生成困难的负面范围外数据用于意图分类

Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent  Classification

Training accurate intent classifiers requires labeled data, which can be
costly to obtain. Data augmentation methods may ameliorate this issue, but the
quality of the generated data varies significantly across techniques. We study
the process of systematically producing pseudo-labeled data given a small seed
set using a wide variety of data augmentation techniques, including mixing
methods together. We find that while certain methods dramatically improve
qualitative and quantitative performance, other methods have minimal or even
negative impact. We also analyze key considerations when implementing data
augmentation methods in production.

研究使用各种数据增强技术来系统地生成伪标记数据的过程，并分析了在生产中实现数据增强方法时的关键考虑因素。

意图分类的数据增强

Data Augmentation for Intent Classification

Intent classifiers are vital to the successful operation of virtual agent
systems. This is especially so in voice activated systems where the data can be
noisy with many ambiguous directions for user intents. Before operation begins,
these classifiers are generally lacking in real-world training data. Active
learning is a common approach used to help label large amounts of collected
user input. However, this approach requires many hours of manual labeling work.
We present the Nearest Neighbors Scores Improvement (NNSI) algorithm for
automatic data selection and labeling. The NNSI reduces the need for manual
labeling by automatically selecting highly-ambiguous samples and labeling them
with high accuracy. This is done by integrating the classifier's output from a
semantically similar group of text samples. The labeled samples can then be
added to the training set to improve the accuracy of the classifier. We
demonstrated the use of NNSI on two large-scale, real-life voice conversation
systems. Evaluation of our results showed that our method was able to select
and label useful samples with high accuracy. Adding these new samples to the
training data significantly improved the classifiers and reduced error rates by
up to 10%.

本文介绍了一种用于自动数据选择和标记的 NNSI 算法，该算法通过集成分类器从语义上相似的文本样本组的输出，自动选择高度模糊的样本并以高精度标记它们。我们在两个大型语音会话系统上演示了 NNSI 的使用，并评估了结果，证明了该方法能够选择并标记高质量的有用样本，加入训练数据集可使分类器的准确性显著提高并将误差率降低高达 10%。