Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining. In this paper, we propose a continual learning benchmark for task-oriented dialogue systems with 37 domains to be learned continuously in four settings, such as intent recognition, state tracking, natural language generation, and end-to-end. Moreover, we implement and compare multiple existing continual learning baselines, and we propose a simple yet effective architectural method based on residual adapters. Our experiments demonstrate that the proposed architectural method and a simple replay-based strategy perform comparably well but they both achieve inferior performance to the multi-task learning baseline, in where all the data are shown at once, showing that continual learning in task-oriented dialogue systems is a challenging task. Furthermore, we reveal several trade-offs between different continual learning methods in term of parameter usage and memory size, which are important in the design of a task-oriented dialogue system. The proposed benchmark is released together with several baselines to promote more research in this direction.

本文提出了一个能够让我们通过时间添加新的领域和功能而不需要付出整个系统重新训练的任务导向对话系统的连续学习基准。我们在四个不同的设置下连续地学习了 37 个领域，包括意图识别、状态跟踪、自然语言生成和端到端。此外，我们实现并比较了多个已有的持续学习基线，并提出了一种简单而有效的基于残余适配器的结构方法。我们的实验证明，所提出的结构方法和简单的回放策略表现相当好，但是它们都比多任务学习基线表现差，后者将所有数据一次性展示出来，这表明对话系统的连续学习是一项具有挑战性的任务。此外，我们还揭示了在参数使用和内存大小方面不同连续学习方法之间的几个权衡，这对任务导向对话系统的设计非常重要。我们还发布了所提出的基准测试以及几个基线，以推动更多的研究方向。

任务导向对话系统中的持续学习