Multimodal language generation, which leverages the synergy of language and
vision, is a rapidly expanding field. However, existing vision-language models
face challenges in tasks that require complex linguistic understanding. To
address this issue, we introduce Visual-Language models as Importance Sampling
weights (VLIS), a novel framework that combines the visual conditioning
capability of vision-language models with the language understanding of
unimodal text-only language models without further training. It extracts
pointwise mutual information of each image and text from a visual-language
model and uses the value as an importance sampling weight to adjust the token
likelihood from a text-only model. VLIS improves vision-language models on
diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and
ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning,
and ROCStories). Our results suggest that VLIS represents a promising new
direction for multimodal language generation.

多模态语言生成领域中，我们引入了 VLIS，这是一个新的框架，将视觉语言模型的视觉调节能力与纯文本语言模型的语言理解相结合，通过提取图像和文本之间的点对互信息，并将其作为重要性抽样权重来调整基于文本的模型的标记生成概率，从而在共识理解和复杂文本生成任务上提升了视觉语言模型的性能。VLIS 代表了多模态语言生成的一个有前途的新方向。