Incorporating multi-modal contexts in conversation is important for
developing more engaging dialogue systems. In this work, we explore this
direction by introducing MMChat: a large-scale Chinese multi-modal dialogue
corpus (32.4M raw dialogues and 120.84K filtered dialogues). Unlike previous
corpora that are crowd-sourced or collected from fictitious movies, MMChat
contains image-grounded dialogues collected from real conversations on social
media, in which the sparsity issue is observed. Specifically, image-initiated
dialogues in common communications may deviate to some non-image-grounded
topics as the conversation proceeds. To better investigate this issue, we
manually annotate 100K dialogues from MMChat and further filter the corpus
accordingly, which yields MMChat-hf. We develop a benchmark model to address
the sparsity issue in dialogue generation tasks by adapting the attention
routing mechanism on image features. Experiments demonstrate the usefulness of
incorporating image features and the effectiveness of handling the sparsity of
image features.

探索利用图片等多模态语境来改善对话系统，使用自然语言数据集 MMChat，通过人工筛选和过滤得到 MMChat-hf，利用注意力机制改善生成任务的稀疏性，结果表明引入图片特征的有效性和处理稀疏性的效力。