Neural generative models have been become increasingly popular when building conversational agents. They offer flexibility, can be easily adapted to new domains, and require minimal domain engineering. A common criticism of these systems is that they seldom understand or use the available dialog history effectively. In this paper, we take an empirical approach to understanding how these models use the available dialog history by studying the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that commonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diagnostic tool for evaluating dialog systems in the future.

本文研究了神经生成模型在对话系统中对历史对话的敏感性，并通过实验证明在测试时的上下文人为干扰（例如缺失或重排序发言，混淆词语等）对常用的递归和基于Transformer的seq2seq模型的影响很小，在开源其代码的同时，本文所提供的测试评估工具将为未来的对话系统改进提供参考。

神经对话系统有效利用对话历史吗？实证研究