Object hallucination poses a significant challenge in vision-language (VL)
models, often leading to the generation of nonsensical or unfaithful responses
with non-existent objects. However, the absence of a general measurement for
evaluating object hallucination in VL models has hindered our understanding and
ability to mitigate this issue. In this work, we present NOPE (Negative Object
Presence Evaluation), a novel benchmark designed to assess object hallucination
in VL models through visual question answering (VQA). We propose a
cost-effective and scalable approach utilizing large language models to
generate 29.5k synthetic negative pronoun (NegP) data of high quality for NOPE.
We extensively investigate the performance of 10 state-of-the-art VL models in
discerning the non-existence of objects in visual questions, where the ground
truth answers are denoted as NegP (e.g., "none"). Additionally, we evaluate
their standard performance on visual questions on 9 other VQA datasets. Through
our experiments, we demonstrate that no VL model is immune to the vulnerability
of object hallucination, as all models achieve accuracy below 10\% on NegP.
Furthermore, we uncover that lexically diverse visual questions, question types
with large scopes, and scene-relevant objects capitalize the risk of object
hallucination in VL models.

基于视觉问题回答的负面对象存在评估（NOPE）是一个新型基准，用于通过大规模语言模型生成高质量的负面代词数据来评估视觉语言模型中的对象幻觉问题。通过对 10 个最先进的视觉语言模型在判断视觉问题中不存在对象的性能进行广泛研究，我们发现没有一个视觉语言模型能免受对象幻觉的影响，因为所有模型在负面代词上的准确率都低于 10%。此外，我们发现在视觉问题中具有词汇多样性、大范围的问题类型和与场景相关的对象会增加视觉语言模型中对象幻觉的风险。

负面目标存在评估（NOPE）：衡量视觉 - 语言模型中的目标幻觉

Negative Object Presence Evaluation (NOPE) to Measure Object  Hallucination in Vision-Language Models

Length generalization, the ability to generalize from small training context
sizes to larger ones, is a critical challenge in the development of
Transformer-based language models. Positional encoding (PE) has been identified
as a major factor influencing length generalization, but the exact impact of
different PE schemes on extrapolation in downstream tasks remains unclear. In
this paper, we conduct a systematic empirical study comparing the length
generalization performance of decoder-only Transformers with five different
position encoding approaches including Absolute Position Embedding (APE), T5's
Relative PE, ALiBi, and Rotary, in addition to Transformers without positional
encoding (NoPE). Our evaluation encompasses a battery of reasoning and
mathematical tasks. Our findings reveal that the most commonly used positional
encoding methods, such as ALiBi, Rotary, and APE, are not well suited for
length generalization in downstream tasks. More importantly, NoPE outperforms
other explicit positional encoding methods while requiring no additional
computation. We theoretically demonstrate that NoPE can represent both absolute
and relative PEs, but when trained with SGD, it mostly resembles T5's relative
PE attention patterns. Finally, we find that scratchpad is not always helpful
to solve length generalization and its format highly impacts the model's
performance. Overall, our work suggests that explicit position embeddings are
not essential for decoder-only Transformers to generalize well to longer
sequences.

本文研究了基于解码器的 Transformer 模型在用不同的位置编码方式时对长度泛化的影响，发现在一系列的推理和数学任务中，NoPE 的表现比其它方法更为优秀，而且无需额外计算。理论上，NoPE 能够代表绝对和相对位置嵌入，但在使用 SGD 训练时，它主要呈现 T5 相对位置嵌入的注意力模式。同时，scratchpad 并不总是有助于解决长度泛化问题，其格式对模型的性能有很大的影响。总之，本文的工作表明解码器 - only 的 Transformer 不一定需要显式的位置嵌入以在更长的序列上泛化良好。