Position bias has proven to be a prevalent issue of modern language models
(LMs), where the models prioritize content based on its position within the
given context. This bias often leads to unexpected model failures and hurts
performance, robustness, and reliability across various applications. Our
mechanistic analysis attributes the position bias to two components employed in
nearly all state-of-the-art LMs: causal attention and relative positional
encodings. Specifically, we find that causal attention generally causes models
to favor distant content, while relative positional encodings like RoPE prefer
nearby ones based on the analysis of retrieval-augmented question answering
(QA). Further, our empirical study on object detection reveals that position
bias is also present in vision-language models (VLMs).
Based on the above analyses, we propose to ELIMINATE position bias caused by
different input segment orders (e.g., options in LM-as-a-judge, retrieved
documents in QA) in a TRAINING-FREE ZERO-SHOT manner. Our method changes the
causal attention to bidirectional attention between segments and utilizes model
attention values to decide the relative orders of segments instead of using the
order provided in input prompts, therefore enabling Position-INvariant
inferencE (PINE) at the segment level. By eliminating position bias, models
achieve better performance and reliability in downstream tasks where position
bias widely exists, such as LM-as-a-judge and retrieval-augmented QA.
Notably, PINE is especially useful when adapting LMs for evaluating reasoning
pairs: it consistently provides 8 to 10 percentage points performance gains in
most cases, and makes Llama-3-70B-Instruct perform even better than
GPT-4-0125-preview on the RewardBench reasoning subset.

通过消除输入段落顺序的不同，我们的方法改变了因果注意力，使其变为段落之间的双向关注，并利用模型关注值来决定段落的相对顺序，从而实现了段落级的位置不变推理（PINE），消除位置偏差使得模型在广泛存在位置偏差的下游任务中获得更好的性能和可靠性。

消除语言模型的位置偏见：一种机制化方法

Eliminating Position Bias of Language Models: A Mechanistic Approach

The distinguishing power of graph transformers is closely tied to the choice
of positional encoding: features used to augment the base transformer with
information about the graph. There are two primary types of positional
encoding: absolute positional encodings (APEs) and relative positional
encodings (RPEs). APEs assign features to each node and are given as input to
the transformer. RPEs instead assign a feature to each pair of nodes, e.g.,
graph distance, and are used to augment the attention block. A priori, it is
unclear which method is better for maximizing the power of the resulting graph
transformer. In this paper, we aim to understand the relationship between these
different types of positional encodings. Interestingly, we show that graph
transformers using APEs and RPEs are equivalent in terms of distinguishing
power. In particular, we demonstrate how to interchange APEs and RPEs while
maintaining their distinguishing power in terms of graph transformers. Based on
our theoretical results, we provide a study on several APEs and RPEs (including
the resistance distance and the recently introduced stable and expressive
positional encoding (SPE)) and compare their distinguishing power in terms of
transformers. We believe our work will help navigate the huge number of choices
of positional encoding and will provide guidance on the future design of
positional encodings for graph transformers.

图转换器的区分能力与位置编码的选择紧密相关，本文旨在理解绝对位置编码和相对位置编码之间的关系，并展示了在图转换器中如何交换这两种类型的位置编码而保持它们的区分能力。基于理论结果，我们对几种位置编码进行了研究并比较了它们在转换器中的区分能力，从而为位置编码的未来设计提供指导。