Image-to-text generation aims to describe images using natural language.
Recently, zero-shot image captioning based on pre-trained vision-language
models (VLMs) and large language models (LLMs) has made significant progress.
However, we have observed and empirically demonstrated that these methods are
susceptible to modality bias induced by LLMs and tend to generate descriptions
containing objects (entities) that do not actually exist in the image but
frequently appear during training (i.e., object hallucination). In this paper,
we propose ViECap, a transferable decoding model that leverages entity-aware
decoding to generate descriptions in both seen and unseen scenarios. ViECap
incorporates entity-aware hard prompts to guide LLMs' attention toward the
visual entities present in the image, enabling coherent caption generation
across diverse scenes. With entity-aware hard prompts, ViECap is capable of
maintaining performance when transferring from in-domain to out-of-domain
scenarios. Extensive experiments demonstrate that ViECap sets a new
state-of-the-art cross-domain (transferable) captioning and performs
competitively in-domain captioning compared to previous VLMs-based zero-shot
methods. Our code is available at: this https URL

我们提出了 ViECap，一种可转移的解码模型，利用实体感知解码生成见过和没见过的场景中的描述。ViECap 通过实体感知的硬提示将 LLMs 的注意力引导到图像中存在的视觉实体，实现了跨多样场景的连贯字幕生成。在通过实体感知的硬提示的情况下，ViECap 能够在从域内到域外的场景转移中保持性能。大量实验证明了 ViECap 在跨域（可转移）字幕生成方面的最新水平，并与以前基于 VLMs 的零样本方法在域内字幕生成方面有竞争力。