Textual response generation is an essential task for multimodal task-oriented
dialog systems.Although existing studies have achieved fruitful progress, they
still suffer from two critical limitations: 1) focusing on the attribute
knowledge but ignoring the relation knowledge that can reveal the correlations
between different entities and hence promote the response generation}, and 2)
only conducting the cross-entropy loss based output-level supervision but
lacking the representation-level regularization. To address these limitations,
we devise a novel multimodal task-oriented dialog system (named MDS-S2).
Specifically, MDS-S2 first simultaneously acquires the context related
attribute and relation knowledge from the knowledge base, whereby the
non-intuitive relation knowledge is extracted by the n-hop graph walk.
Thereafter, considering that the attribute knowledge and relation knowledge can
benefit the responding to different levels of questions, we design a
multi-level knowledge composition module in MDS-S2 to obtain the latent
composed response representation. Moreover, we devise a set of latent query
variables to distill the semantic information from the composed response
representation and the ground truth response representation, respectively, and
thus conduct the representation-level semantic regularization. Extensive
experiments on a public dataset have verified the superiority of our proposed
MDS-S2. We have released the codes and parameters to facilitate the research
community.

提出了一种名为 MDS-S2 的新型多模式任务导向对话系统，通过从知识库中获取相关属性和关系知识，设计了多级知识组成模块以及一组潜在查询变量以实现表征级别的语义规范化来解决文本响应生成过程中的关键问题。

双语义知识组合的多模态对话系统

Dual Semantic Knowledge Composed Multimodal Dialog Systems

Text response generation for multimodal task-oriented dialog systems, which
aims to generate the proper text response given the multimodal context, is an
essential yet challenging task. Although existing efforts have achieved
compelling success, they still suffer from two pivotal limitations: 1) overlook
the benefit of generative pre-training, and 2) ignore the textual context
related knowledge. To address these limitations, we propose a novel dual
knowledge-enhanced generative pretrained language model for multimodal
task-oriented dialog systems (DKMD), consisting of three key components: dual
knowledge selection, dual knowledge-enhanced context learning, and
knowledge-enhanced response generation. To be specific, the dual knowledge
selection component aims to select the related knowledge according to both
textual and visual modalities of the given context. Thereafter, the dual
knowledge-enhanced context learning component targets seamlessly integrating
the selected knowledge into the multimodal context learning from both global
and local perspectives, where the cross-modal semantic relation is also
explored. Moreover, the knowledge-enhanced response generation component
comprises a revised BART decoder, where an additional dot-product
knowledge-decoder attention sub-layer is introduced for explicitly utilizing
the knowledge to advance the text response generation. Extensive experiments on
a public dataset verify the superiority of the proposed DKMD over
state-of-the-art competitors.

本文提出了一个可以双重增强知识的生成式语言模型，用于为多模态任务导向式对话系统生成适当的文本响应。实验证明，该模型优于现有竞争对手。