This paper presents an ontology-aware pretrained language model (OPAL) for
end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models,
task-oriented dialogue models fulfill at least two task-specific modules:
dialogue state tracker (DST) and response generator (RG). The dialogue state
consists of the domain-slot-value triples, which are regarded as the user's
constraints to search the domain-related databases. The large-scale
task-oriented dialogue data with the annotated structured dialogue state
usually are inaccessible. It prevents the development of the pretrained
language model for the task-oriented dialogue. We propose a simple yet
effective pretraining method to alleviate this problem, which consists of two
pretraining phases. The first phase is to pretrain on large-scale contextual
text data, where the structured information of the text is extracted by the
information extracting tool. To bridge the gap between the pretraining method
and downstream tasks, we design two pretraining tasks: ontology-like triple
recovery and next-text generation, which simulates the DST and RG,
respectively. The second phase is to fine-tune the pretrained model on the TOD
data. The experimental results show that our proposed method achieves an
exciting boost and get competitive performance even without any TOD data on
CamRest676 and MultiWOZ benchmarks.

本文提出了一种面向任务的对话技术（TOD）前置训练语言模型 (OPAL)，采用预训练和微调相结合的方法，成功地解决了无法获得大规模任务对话数据的问题，并在 CamRest676 和 MultiWOZ 基准测试中实现了令人兴奋的提升和竞争性表现。