Large language models (LLMs) have shown remarkable performance in natural
language processing (NLP) tasks. To comprehend and execute diverse human
instructions over image data, instruction-tuned large vision-language models
(LVLMs) have been introduced. However, LVLMs may suffer from different types of
object hallucinations. Nevertheless, LVLMs are evaluated for coarse-grained
object hallucinations only (i.e., generated objects non-existent in the input
image). The fine-grained object attributes and behaviors non-existent in the
image may still be generated but not measured by the current evaluation
methods. In this paper, we thus focus on reducing fine-grained hallucinations
of LVLMs. We propose \textit{ReCaption}, a framework that consists of two
components: rewriting captions using ChatGPT and fine-tuning the
instruction-tuned LVLMs on the rewritten captions. We also propose a
fine-grained probing-based evaluation method named \textit{Fine-Grained Object
Hallucination Evaluation} (\textit{FGHE}). Our experiment results demonstrate
that ReCaption effectively reduces fine-grained object hallucination for
different LVLM options and improves their text generation quality. The code can
be found at this https URL

本文介绍了 ReCaption, 一个用于减少视觉 - 语言模型中细粒度物体幻觉的框架，并通过实验证明其有效性。