Zero-shot image captioning (IC) without well-paired image-text data can be
divided into two categories, training-free and text-only-training. Generally,
these two types of methods realize zero-shot IC by integrating pretrained
vision-language models like CLIP for image-text similarity evaluation and a
pre-trained language model (LM) for caption generation. The main difference
between them is whether using a textual corpus to train the LM. Though
achieving attractive performance w.r.t. some metrics, existing methods often
exhibit some common drawbacks. Training-free methods tend to produce
hallucinations, while text-only-training often lose generalization capability.
To move forward, in this paper, we propose a novel Memory-Augmented zero-shot
image Captioning framework (MeaCap). Specifically, equipped with a textual
memory, we introduce a retrieve-then-filter module to get key concepts that are
highly related to the image. By deploying our proposed memory-augmented
visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate
concept-centered captions that keep high consistency with the image with fewer
hallucinations and more world-knowledge. The framework of MeaCap achieves the
state-of-the-art performance on a series of zero-shot IC settings. Our code is
available at this https URL

提出了一种新颖的记忆增强型零样本图像字幕生成框架（MeaCap），通过装备文本记忆并引入检索 - 过滤模块，使用基于记忆的视觉相关融合评分及关键词 - 句子语言模型，生成与图像高度一致、拥有更少幻觉和更多世界知识的以概念为中心的字幕；该框架在一系列零样本图像字幕设置中取得了最先进的性能。