Retrieval Augmented Generation (RAG) is widely employed to ground responses
to queries on domain-specific documents. But do RAG implementations leave out
important information or excessively include irrelevant information? To allay
these concerns, it is necessary to annotate domain-specific benchmarks to
evaluate information retrieval (IR) performance, as relevance definitions vary
across queries and domains. Furthermore, such benchmarks should be
cost-efficiently annotated to avoid annotation selection bias. In this paper,
we propose DIRAS (Domain-specific Information Retrieval Annotation with
Scalability), a manual-annotation-free schema that fine-tunes open-sourced LLMs
to annotate relevance labels with calibrated relevance probabilities. Extensive
evaluation shows that DIRAS fine-tuned models achieve GPT-4-level performance
on annotating and ranking unseen (query, document) pairs, and is helpful for
real-world RAG development.

本文提出了 DIRAS（Domain-specific Information Retrieval Annotation with Scalability）方法，通过细调开源 LLMs 来使用校准的相关性概率注释相关标签，实现了 GPT-4 级别的模型性能，对于实际的 RAG 开发具有帮助。

DIRAS：检索增强生成中高效的 LLM 辅助文档相关性标注

DIRAS: Efficient LLM-Assisted Annotation of Document Relevance in  Retrieval Augmented Generation

Relevance labels, which indicate whether a search result is valuable to a
searcher, are key to evaluating and optimising search systems. The best way to
capture the true preferences of users is to ask them for their careful feedback
on which results would be useful, but this approach does not scale to produce a
large number of labels. Getting relevance labels at scale is usually done with
third-party labellers, who judge on behalf of the user, but there is a risk of
low-quality data if the labeller doesn't understand user needs. To improve
quality, one standard approach is to study real users through interviews, user
studies and direct feedback, find areas where labels are systematically
disagreeing with users, then educate labellers about user needs through judging
guidelines, training and monitoring. This paper introduces an alternate
approach for improving label quality. It takes careful feedback from real
users, which by definition is the highest-quality first-party gold data that
can be derived, and develops an large language model prompt that agrees with
that data.
We present ideas and observations from deploying language models for
large-scale relevance labelling at Bing, and illustrate with data from TREC. We
have found large language models can be effective, with accuracy as good as
human labellers and similar capability to pick the hardest queries, best runs,
and best groups. Systematic changes to the prompts make a difference in
accuracy, but so too do simple paraphrases. To measure agreement with real
searchers needs high-quality ``gold'' labels, but with these we find that
models produce better labels than third-party workers, for a fraction of the
cost, and these labels let us train notably better rankers.

通过获取真实用户的仔细反馈来改进相关性标签的质量，并提出一种基于大型语言模型的标签生成方法，从而获得具有和人工标记员相似的准确性和能力的高质量标签，为搜索系统的评估和优化提供了有效途径。

大型语言模型能准确预测搜索者偏好

Large language models can accurately predict searcher preferences

This work proposes a novel adaptation of a pretrained sequence-to-sequence
model to the task of document ranking. Our approach is fundamentally different
from a commonly-adopted classification-based formulation of ranking, based on
encoder-only pretrained transformer architectures such as BERT. We show how a
sequence-to-sequence model can be trained to generate relevance labels as
"target words", and how the underlying logits of these target words can be
interpreted as relevance probabilities for ranking. On the popular MS MARCO
passage ranking task, experimental results show that our approach is at least
on par with previous classification-based models and can surpass them with
larger, more-recent models. On the test collection from the TREC 2004 Robust
Track, we demonstrate a zero-shot transfer-based approach that outperforms
previous state-of-the-art models requiring in-dataset cross-validation.
Furthermore, we find that our approach significantly outperforms an
encoder-only model in a data-poor regime (i.e., with few training examples). We
investigate this observation further by varying target words to probe the
model's use of latent knowledge.

通过对预训练序列到序列模型进行新的调整，使其能够用于文档排名任务，相比较于 BERT 等仅采用编码器预训练变压器架构的基于分类的排名方法，我们的方法在实验中展现出了更好的性能，并发现我们的方法在数据较少的情况下更能胜任。