Despite achieving outstanding performance on various cross-modal tasks,
current large vision-language models (LVLMs) still suffer from hallucination
issues, manifesting as inconsistencies between their generated responses and
the corresponding images. Prior research has implicated that the low quality of
instruction data, particularly the skewed balance between positive and negative
samples, is a significant contributor to model hallucinations. Recently,
researchers have proposed high-quality instruction datasets, such as
LRV-Instruction, to mitigate model hallucination. Nonetheless, our
investigation reveals that hallucinatory concepts from different LVLMs exhibit
specificity, i.e. the distribution of hallucinatory concepts varies
significantly across models. Existing datasets did not consider the
hallucination specificity of different models in the design processes, thereby
diminishing their efficacy in mitigating model hallucination. In this paper, we
propose a targeted instruction data generation framework named DFTG that
tailored to the hallucination specificity of different models. Concretely, DFTG
consists of two stages: hallucination diagnosis, which extracts the necessary
information from the model's responses and images for hallucination diagnosis;
and targeted data generation, which generates targeted instruction data based
on diagnostic results. The experimental results on hallucination benchmarks
demonstrate that the targeted instruction data generated by our method are more
effective in mitigating hallucinations compared to previous datasets.

针对当前大型视觉 - 语言模型在生成回复和相应图像之间存在的不一致问题，本研究提出了一种名为 DFTG 的有针对性的指导数据生成框架，通过诊断模型的响应和图像，生成有针对性的指导数据以减轻幻觉问题。实验结果表明，与先前的数据集相比，我们的方法生成的有针对性指导数据在减轻幻觉方面更加有效。