Image-grounded dialogue systems benefit greatly from integrating visual
information, resulting in high-quality response generation. However, current
models struggle to effectively utilize such information in zero-resource
scenarios, mainly due to the disparity between image and text modalities. To
overcome this challenge, we propose an innovative multimodal framework, called
ZRIGF, which assimilates image-grounded information for dialogue generation in
zero-resource situations. ZRIGF implements a two-stage learning strategy,
comprising contrastive pre-training and generative pre-training. Contrastive
pre-training includes a text-image matching module that maps images and texts
into a unified encoded vector space, along with a text-assisted masked image
modeling module that preserves pre-training visual features and fosters further
multimodal feature alignment. Generative pre-training employs a multimodal
fusion module and an information transfer module to produce insightful
responses based on harmonized multimodal representations. Comprehensive
experiments conducted on both text-based and image-grounded dialogue datasets
demonstrate ZRIGF's efficacy in generating contextually pertinent and
informative responses. Furthermore, we adopt a fully zero-resource scenario in
the image-grounded dialogue dataset to demonstrate our framework's robust
generalization capabilities in novel domains. The code is available at
this https URL

ZRIGF 是一种创新的多模态框架，用于零资源情境下基于图像的对话生成，包括对抗性预训练和生成性预训练两个阶段，通过映射和融合图像和文本，生成上下文相关和有信息量的回应。