Large vision-language models (LVLMs) have significantly improved multimodal
reasoning tasks, such as visual question answering and image captioning. These
models embed multimodal facts within their parameters, rather than relying on
external knowledge bases to store factual information explicitly. However, the
content discerned by LVLMs may deviate from actual facts due to inherent bias
or incorrect inference. To address this issue, we introduce MFC-Bench, a
rigorous and comprehensive benchmark designed to evaluate the factual accuracy
of LVLMs across three tasks: Manipulation, Out-of-Context, and Veracity
Classification. Through our evaluation on MFC-Bench, we benchmarked 12 diverse
and representative LVLMs, uncovering that current models still fall short in
multimodal fact-checking and demonstrate insensitivity to various forms of
manipulated content. We hope that MFC-Bench could raise attention to the
trustworthy artificial intelligence potentially assisted by LVLMs in the
future. The MFC-Bench and accompanying resources are publicly accessible at
this https URL, contributing to ongoing research in the
multimodal fact-checking field.

通过评估 MFC-Bench 对 12 个各异而代表性的大型视觉 - 语言模型进行了基准测试，发现当前模型在多模态事实核查方面仍存在不足，并对各种形式的操纵内容表现出麻木，希望通过 MFC-Bench 能够引起对未来可能由大型视觉 - 语言模型辅助的可信人工智能的关注。

MFC-Bench: 基于大型视觉 - 语言模型的多模态事实核查基准测试

MFC-Bench: Benchmarking Multimodal Fact-Checking with Large  Vision-Language Models

Recent advances in vision-and-language modeling have seen the development of
Transformer architectures that achieve remarkable performance on multimodal
reasoning tasks. Yet, the exact capabilities of these black-box models are
still poorly understood. While much of previous work has focused on studying
their ability to learn meaning at the word-level, their ability to track
syntactic dependencies between words has received less attention. We take a
first step in closing this gap by creating a new multimodal task targeted at
evaluating understanding of predicate-noun dependencies in a controlled setup.
We evaluate a range of state-of-the-art models and find that their performance
on the task varies considerably, with some models performing relatively well
and others at chance level. In an effort to explain this variability, our
analyses indicate that the quality (and not only sheer quantity) of pretraining
data is essential. Additionally, the best performing models leverage
fine-grained multimodal pretraining objectives in addition to the standard
image-text matching objectives. This study highlights that targeted and
controlled evaluations are a crucial step for a precise and rigorous test of
the multimodal knowledge of vision-and-language models.

本文研究视觉 - 语言建模，通过创建新的多模态任务和分析预训练数据的质量，发现预训练数据的质量和多模态预训练目标对模型的性能影响重要。