Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both based on the dialogue context. Due to the lack of a large-scale dataset specifically for this task and the benefits of leveraging powerful pre-trained models, previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. However, this approach can overlook crucial information about the image, hindering 1) image-grounded text response and 2) consistency of objects in the image response. In this paper, we propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content and the consistency of objects in sequential image responses. Through extensive experiments on the multimodal dialogue benchmark dataset, we show that BI-MDRG can effectively increase the quality of multimodal dialogue. Additionally, recognizing the gap in benchmark datasets for evaluating the image consistency in multimodal dialogue, we have created a curated set of 300 dialogues annotated to track object consistency across conversations.

本研究解决了多模态对话响应生成任务中缺乏大规模数据集导致的信息遗漏问题。提出的BI-MDRG方法通过利用图像历史信息，提高了文本响应与图像内容的相关性及图像响应中对象的一致性。研究结果显示BI-MDRG显著提升了多模态对话的质量，并创建了一个300个对话的标注数据集以评估图像一致性。

BI-MDRG：在多模态对话响应生成中桥接图像历史