Hallucinations and unfaithful synthesis due to inaccurate prompts with
insufficient semantic details are widely observed in multimodal generative
models. A prevalent strategy to align multiple modalities is to fine-tune the
generator with a large number of annotated text-image pairs. However, such a
procedure is labor-consuming and resource-draining. The key question we ask is:
can we enhance the quality and faithfulness of text-driven generative models
beyond extensive text-image pair annotations? To address this question, we
propose Knowledge Pursuit Prompting (KPP), a zero-shot framework that
iteratively incorporates external knowledge to help generators produce reliable
visual content. Instead of training generators to handle generic prompts, KPP
employs a recursive knowledge query process to gather informative external
facts from the knowledge base, instructs a language model to compress the
acquired knowledge for prompt refinement, and utilizes text-driven generators
for visual synthesis. The entire process is zero-shot, without accessing the
architectures and parameters of generative models. We evaluate the framework
across multiple text-driven generative tasks (image, 3D rendering, and video)
on datasets of different domains. We further demonstrate the extensibility and
adaptability of KPP through varying foundation model bases and instructions.
Our results show that KPP is capable of generating faithful and semantically
rich content across diverse visual domains, offering a promising solution to
improve multimodal generative models.

通过迭代地整合外部知识以帮助生成模型产生可靠的视觉内容，无需广泛的文本 - 图像对注释，Knowledge Pursuit Prompting (KPP) 是一种零样本框架，用于提高多模态生成模型的质量和忠诚度。