Information Retrieval (IR) systems are designed to deliver relevant content,
but traditional systems may not optimize rankings for fairness, neutrality, or
the balance of ideas. Consequently, IR can often introduce indexical biases, or
biases in the positional order of documents. Although indexical bias can
demonstrably affect people's opinion, voting patterns, and other behaviors,
these issues remain understudied as the field lacks reliable metrics and
procedures for automatically measuring indexical bias. Towards this end, we
introduce the PAIR framework, which supports automatic bias audits for ranked
documents or entire IR systems. After introducing DUO, the first
general-purpose automatic bias metric, we run an extensive evaluation of 8 IR
systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k
queries spanning 1.4k controversial issue topics. A human behavioral study
validates our approach, showing that our bias metric can help predict when and
how indexical bias will shift a reader's opinion.

信息检索系统的相关性排序可能引入指示偏见，该研究引入了自动偏见审核框架（PAIR framework）和偏见度量（bias metric），通过对 8 个信息检索系统进行评估，在一系列有争议的主题中，验证了指示偏见如何影响读者意见的预测。