Most language understanding models in task-oriented dialog systems are
trained on a small amount of annotated training data, and evaluated in a small
set from the same distribution. However, these models can lead to system
failure or undesirable output when being exposed to natural language
perturbation or variation in practice. In this paper, we conduct comprehensive
evaluation and analysis with respect to the robustness of natural language
understanding models, and introduce three important aspects related to language
understanding in real-world dialog systems, namely, language variety, speech
characteristics, and noise perturbation. We propose a model-agnostic toolkit
LAUG to approximate natural language perturbations for testing the robustness
issues in task-oriented dialog. Four data augmentation approaches covering the
three aspects are assembled in LAUG, which reveals critical robustness issues
in state-of-the-art models. The augmented dataset through LAUG can be used to
facilitate future research on the robustness testing of language understanding
in task-oriented dialog.

本研究针对自然语言理解模型在实际对话系统的应用中容易出现的波动和变化问题，提出了一种模型无关的工具箱 LAUG，涵盖语言变体，语音特性和噪声扰动三个方面的四种数据增强方法，揭示了现有模型中的严重鲁棒性问题，提供了一种使用 LAUG 生成的增强数据集来促进语言理解测试鲁棒性的方法。

面向任务对话的语言理解鲁棒性测试

Robustness Testing of Language Understanding in Task-Oriented Dialog

We describe our participation in the PAN 2017 shared task on Author
Profiling, identifying authors' gender and language variety for English,
Spanish, Arabic and Portuguese. We describe both the final, submitted system,
and a series of negative results. Our aim was to create a single model for both
gender and language, and for all language varieties. Our best-performing system
(on cross-validated results) is a linear support vector machine (SVM) with word
unigrams and character 3- to 5-grams as features. A set of additional features,
including POS tags, additional datasets, geographic entities, and Twitter
handles, hurt, rather than improve, performance. Results from cross-validation
indicated high performance overall and results on the test set confirmed them,
at 0.86 averaged accuracy, with performance on sub-tasks ranging from 0.68 to
0.98.

文章描述了我们参加 PAN 2017 的作者识别竞赛，其中我们使用支持向量机作为算法进行性别和语言变量的识别。最终的结果表明，我们使用交叉验证的方法，成功地将算法的平均准确率提高到了 0.86，证明了算法的有效性。