Large Vision-Language Models (LVLMs) have demonstrated outstanding
performance across various multimodal tasks. However, they suffer from a
problem known as language prior, where responses are generated based solely on
textual patterns while disregarding image information. Addressing the issue of
language prior is crucial, as it can lead to undesirable biases or
hallucinations when dealing with images that are out of training distribution.
Despite its importance, current methods for accurately measuring language
priors in LVLMs are poorly studied. Although existing benchmarks based on
counterfactual or out-of-distribution images can partially be used to measure
language priors, they fail to disentangle language priors from other
confounding factors. To this end, we propose a new benchmark called
VLind-Bench, which is the first benchmark specifically designed to measure the
language priors, or blindness, of LVLMs. It not only includes tests on
counterfactual images to assess language priors but also involves a series of
tests to evaluate more basic capabilities such as commonsense knowledge, visual
perception, and commonsense biases. For each instance in our benchmark, we
ensure that all these basic tests are passed before evaluating the language
priors, thereby minimizing the influence of other factors on the assessment.
The evaluation and analysis of recent LVLMs in our benchmark reveal that almost
all models exhibit a significant reliance on language priors, presenting a
strong challenge in the field.

通过新的基准测试 VLind-Bench，本研究评估和分析了近期的大型视觉语言模型 (LVLMs)，发现几乎所有模型都过度依赖于语言先验，这对该领域构成了巨大挑战。

VLind-Bench：大型视觉 - 语言模型中的语言先验测量

VLind-Bench: Measuring Language Priors in Large Vision-Language Models

Recent works have shown huge success of deep learning models for common in
vocabulary (IV) scene text recognition. However, in real-world scenarios,
out-of-vocabulary (OOV) words are of great importance and SOTA recognition
models usually perform poorly on OOV settings. Inspired by the intuition that
the learned language prior have limited OOV preformence, we design a framework
named Vision Language Adaptive Mutual Decoder (VLAMD) to tackle OOV problems
partly. VLAMD consists of three main conponents. Firstly, we build an attention
based LSTM decoder with two adaptively merged visual-only modules, yields a
vision-language balanced main branch. Secondly, we add an auxiliary query based
autoregressive transformer decoding head for common visual and language prior
representation learning. Finally, we couple these two designs with
bidirectional training for more diverse language modeling, and do mutual
sequential decoding to get robuster results. Our approach achieved 70.31\% and
59.61\% word accuracy on IV+OOV and OOV settings respectively on Cropped Word
Recognition Task of OOV-ST Challenge at ECCV 2022 TiE Workshop, where we got
1st place on both settings.

该研究提出了一种名为 VLAMD 的框架，该框架基于深度学习，利用语言先验知识，设计了一种针对 OCR 中存在的 Out-of-vocabulary 问题的解决方案，其中包括注意力 LSTM 解码器，自回归转换器解码器和双向训练方法，该方法在 2022 ECCV TiE Workshop 的 OOV-ST 挑战中表现出良好的性能。

为 OOV-STR 设计的视觉语言自适应互补解码器

Vision-Language Adaptive Mutual Decoder for OOV-STR

Despite significant progress has been achieved in text summarization, factual
inconsistency in generated summaries still severely limits its practical
applications. Among the key factors to ensure factual consistency, a reliable
automatic evaluation metric is the first and the most crucial one. However,
existing metrics either neglect the intrinsic cause of the factual
inconsistency or rely on auxiliary tasks, leading to an unsatisfied correlation
with human judgments or increasing the inconvenience of usage in practice. In
light of these challenges, we propose a novel metric to evaluate the factual
consistency in text summarization via counterfactual estimation, which
formulates the causal relationship among the source document, the generated
summary, and the language prior. We remove the effect of language prior, which
can cause factual inconsistency, from the total causal effect on the generated
summary, and provides a simple yet effective way to evaluate consistency
without relying on other auxiliary tasks. We conduct a series of experiments on
three public abstractive text summarization datasets, and demonstrate the
advantages of the proposed metric in both improving the correlation with human
judgments and the convenience of usage. The source code is available at
this https URL

提出一种基于反事实估算的新型度量标准，用于评估文本摘要的事实一致性，能够帮助改善与人类判断的相关性和使用的便利性。