For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities or domains. In pursuit of these goals, we introduce the RADDLE benchmark, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains. By including tasks with limited training data, RADDLE is designed to favor and encourage models with a strong generalization ability. RADDLE also includes a diagnostic checklist that facilitates detailed robustness analysis in aspects such as language variations, speech errors, unseen entities, and out-of-domain utterances. We evaluate recent state-of-the-art systems based on pre-training and fine-tuning, and find that grounded pre-training on heterogeneous dialog corpora performs better than training a separate model per domain. Overall, existing models are less than satisfactory in robustness evaluation, which suggests opportunities for future improvement.

为了最大限度地提高面向任务的对话系统的实用性，该论文引入了 RADDLE 基准，该基准是一个语料库和工具的集合，用于评估模型在不同领域的性能。通过包括训练数据有限的任务，RADDLE 旨在鼓励具有强大泛化能力的模型。通过现有模型的评估，发现它们在鲁棒性评估方面不尽如人意，这提示了未来的改进机会。

RADDLE：一个稳健的面向任务型对话系统的评估基准与分析平台