Information Retrieval (IR) systems are designed to deliver relevant content,
but traditional systems may not optimize rankings for fairness, neutrality, or
the balance of ideas. Consequently, IR can often introduce indexical biases, or
biases in the positional order of documents. Although indexical bias can
demonstrably affect people's opinion, voting patterns, and other behaviors,
these issues remain understudied as the field lacks reliable metrics and
procedures for automatically measuring indexical bias. Towards this end, we
introduce the PAIR framework, which supports automatic bias audits for ranked
documents or entire IR systems. After introducing DUO, the first
general-purpose automatic bias metric, we run an extensive evaluation of 8 IR
systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k
queries spanning 1.4k controversial issue topics. A human behavioral study
validates our approach, showing that our bias metric can help predict when and
how indexical bias will shift a reader's opinion.

信息检索系统的相关性排序可能引入指示偏见，该研究引入了自动偏见审核框架（PAIR framework）和偏见度量（bias metric），通过对 8 个信息检索系统进行评估，在一系列有争议的主题中，验证了指示偏见如何影响读者意见的预测。

信息检索中的索引偏见的测量和应对

Measuring and Addressing Indexical Bias in Information Retrieval

We present the Benchmark of Information Retrieval (IR) tasks with Complex
Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve
documents given multi-faceted user objectives. The benchmark's complexity and
compact size make it suitable for evaluating large language model (LLM)-based
information retrieval systems. We present a modular framework for investigating
factors that may influence LLM performance on retrieval tasks, and identify a
simple baseline model which matches or outperforms existing approaches and more
complex alternatives. No approach achieves satisfactory performance on all
benchmark tasks, suggesting that stronger models and new retrieval protocols
are necessary to address complex user needs.

我们提出了一个信息检索任务的基准（BIRCO），它评估了信息检索系统在给定多方面用户目标的情况下检索文档的能力。该基准的复杂性和紧凑性使其适合评估基于大型语言模型（LLM）的信息检索系统。我们提出了一个模块化框架，用于研究可能影响 LLM 在检索任务上的性能的因素，并识别了一个简单的基准模型，它与现有方法和更复杂的替代方法相匹配或胜过。没有一种方法能够满足所有基准任务的性能要求，这表明需要更强大的模型和新的检索协议来满足复杂的用户需求。

BIRCO：具有复杂目标的信息检索任务基准

BIRCO: A Benchmark of Information Retrieval Tasks with Complex  Objectives

We argue that current IR metrics, modeled on optimizing user experience,
measure too narrow a portion of the IR space. If IR systems are weak, these
metrics undersample or completely filter out the deeper documents that need
improvement. If IR systems are relatively strong, these metrics undersample
deeper relevant documents that could underpin even stronger IR systems, ones
that could present content from tens or hundreds of relevant documents in a
user-digestible hierarchy or text summary. We reanalyze over 70 TREC tracks
from the past 28 years, showing that roughly half undersample top ranked
documents and nearly all undersample tail documents. We show that in the 2020
Deep Learning tracks, neural systems were actually near-optimal at top-ranked
documents, compared to only modest gains over BM25 on tail documents. Our
analysis is based on a simple new systems-oriented metric, 'atomized search
length', which is capable of accurately and evenly measuring all relevant
documents at any depth.

本文研究基于用户体验的现有 IR 度量过于狭窄，如何使用新的系统度量方法来均匀而准确地测量 IR 系统中的所有相关文档，并在超过 70 个 TREC 轨迹以及 2020 年深度学习轨迹中展示了结果。

原子化搜索长度：超越用户模型

Atomized Search Length: Beyond User Models

Machine learning plays a role in many aspects of modern IR systems, and deep
learning is applied in all of them. The fast pace of modern-day research has
given rise to many different approaches for many different IR problems. The
amount of information available can be overwhelming both for junior students
and for experienced researchers looking for new research topics and directions.
Additionally, it is interesting to see what key insights into IR problems the
new technologies are able to give us. The aim of this full-day tutorial is to
give a clear overview of current tried-and-trusted neural methods in IR and how
they benefit IR research. It covers key architectures, as well as the most
promising future directions.

本研究论文旨在向初学者和有经验的研究人员介绍当前 IR 问题的深度学习中的神经方法，并涵盖了关键架构和最有前途的未来方向。