Text response generation for multimodal task-oriented dialog systems, which
aims to generate the proper text response given the multimodal context, is an
essential yet challenging task. Although existing efforts have achieved
compelling success, they still suffer from two pivotal limitations: 1) overlook
the benefit of generative pre-training, and 2) ignore the textual context
related knowledge. To address these limitations, we propose a novel dual
knowledge-enhanced generative pretrained language model for multimodal
task-oriented dialog systems (DKMD), consisting of three key components: dual
knowledge selection, dual knowledge-enhanced context learning, and
knowledge-enhanced response generation. To be specific, the dual knowledge
selection component aims to select the related knowledge according to both
textual and visual modalities of the given context. Thereafter, the dual
knowledge-enhanced context learning component targets seamlessly integrating
the selected knowledge into the multimodal context learning from both global
and local perspectives, where the cross-modal semantic relation is also
explored. Moreover, the knowledge-enhanced response generation component
comprises a revised BART decoder, where an additional dot-product
knowledge-decoder attention sub-layer is introduced for explicitly utilizing
the knowledge to advance the text response generation. Extensive experiments on
a public dataset verify the superiority of the proposed DKMD over
state-of-the-art competitors.

本文提出了一个可以双重增强知识的生成式语言模型，用于为多模态任务导向式对话系统生成适当的文本响应。实验证明，该模型优于现有竞争对手。