Integrating multimodal knowledge into large language models (LLMs) represents
a significant advancement in dialogue generation capabilities. However, the
effective incorporation of such knowledge in zero-resource scenarios remains a
substantial challenge due to the scarcity of diverse, high-quality dialogue
datasets. To address this, we propose the Visual Implicit Knowledge
Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs
for enriched dialogue generation in zero-resource contexts by leveraging
implicit multimodal knowledge. VIKDF comprises two main stages: knowledge
distillation, using an Implicit Query Transformer to extract and encode visual
implicit knowledge from image-text pairs into knowledge vectors; and knowledge
integration, employing a novel Bidirectional Variational Information Fusion
technique to seamlessly integrate these distilled vectors into LLMs. This
enables the LLMs to generate dialogues that are not only coherent and engaging
but also exhibit a deep understanding of the context through implicit
multimodal cues, effectively overcoming the limitations of zero-resource
scenarios. Our extensive experimentation across two dialogue datasets shows
that VIKDF outperforms existing state-of-the-art models in generating
high-quality dialogues. The code will be publicly available following
acceptance.

通过利用隐式多模态知识，将视觉隐式知识融入基于语言模型的对话生成系统，能够在零资源情境下实现更丰富的对话生成，我们提出了一种名为 VIKDF 的框架，通过两个主要阶段（知识蒸馏和知识整合）实现，能够有效解决零资源情境中多样、高质量对话数据集稀缺性所带来的挑战，并取得了优于现有模型的生成对话质量。

将隐式多模态知识融入到零资源对话生成中的 L-LMs

Distilling Implicit Multimodal Knowledge into LLMs for Zero-Resource  Dialogue Generation

Image-grounded dialogue systems benefit greatly from integrating visual
information, resulting in high-quality response generation. However, current
models struggle to effectively utilize such information in zero-resource
scenarios, mainly due to the disparity between image and text modalities. To
overcome this challenge, we propose an innovative multimodal framework, called
ZRIGF, which assimilates image-grounded information for dialogue generation in
zero-resource situations. ZRIGF implements a two-stage learning strategy,
comprising contrastive pre-training and generative pre-training. Contrastive
pre-training includes a text-image matching module that maps images and texts
into a unified encoded vector space, along with a text-assisted masked image
modeling module that preserves pre-training visual features and fosters further
multimodal feature alignment. Generative pre-training employs a multimodal
fusion module and an information transfer module to produce insightful
responses based on harmonized multimodal representations. Comprehensive
experiments conducted on both text-based and image-grounded dialogue datasets
demonstrate ZRIGF's efficacy in generating contextually pertinent and
informative responses. Furthermore, we adopt a fully zero-resource scenario in
the image-grounded dialogue dataset to demonstrate our framework's robust
generalization capabilities in novel domains. The code is available at
this https URL

ZRIGF 是一种创新的多模态框架，用于零资源情境下基于图像的对话生成，包括对抗性预训练和生成性预训练两个阶段，通过映射和融合图像和文本，生成上下文相关和有信息量的回应。