The purpose of modeling document relevance for search engines is to rank
better in subsequent searches. Document-specific historical click-through rates
can be important features in a dynamic ranking system which updates as we
accumulate more sample. This paper describes the properties of several such
features, and tests them in controlled experiments. Extending the inverse
propensity weighting method to documents creates an unbiased estimate of
document relevance. This feature can approximate relevance accurately, leading
to near-optimal ranking in ideal circumstances. However, it has high variance
that is increasing with respect to the degree of position bias. Furthermore,
inaccurate position bias estimation leads to poor performance. Under several
scenarios this feature can perform worse than biased click-through rates. This
paper underscores the need for accurate position bias estimation, and is unique
in suggesting simultaneous use of biased and unbiased position bias features.

本文介绍了排名系统中模拟文档相关性的目的，并测试了几个特征的性质。将逆倾向权重方法用于文档可创建不偏的文档相关性估计特征，该特征可以准确地近似相关性，在理想情况下实现接近最佳的排名。然而，此特征具有较高的方差，且方差随着位置偏差的增加而增加。不准确的位置偏差估计导致性能下降。本文强调了准确估计位置偏差的必要性，并独特地提出了同时使用有偏和无偏位置偏差特征的建议。

特征中的位置偏差

Position bias in features

In the field of information retrieval, Query Likelihood Models (QLMs) rank
documents based on the probability of generating the query given the content of
a document. Recently, advanced large language models (LLMs) have emerged as
effective QLMs, showcasing promising ranking capabilities. This paper focuses
on investigating the genuine zero-shot ranking effectiveness of recent LLMs,
which are solely pre-trained on unstructured text data without supervised
instruction fine-tuning. Our findings reveal the robust zero-shot ranking
ability of such LLMs, highlighting that additional instruction fine-tuning may
hinder effectiveness unless a question generation task is present in the
fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking
system that integrates LLM-based QLMs with a hybrid zero-shot retriever,
demonstrating exceptional effectiveness in both zero-shot and few-shot
scenarios. We make our codebase publicly available at
this https URL

本研究针对最近的大型语言模型探究了原本的零样本排名效果，发现这些仅基于非结构化文本数据进行预训练且没有经过监督指导微调的模型具备强大的零样本排名能力。此外，我们还提出了一个新颖的最先进排名系统，将基于大型语言模型的查询似然模型与混合式零样本检索器相结合，无论是在零样本还是少样本场景下都表现出卓越的效果。我们在 this https URL 上公开了我们的代码库。

开源大型语言模型是用于文档排序的强零射击查询似然模型

Open-source Large Language Models are Strong Zero-shot Query Likelihood  Models for Document Ranking

As the heart of a search engine, the ranking system plays a crucial role in
satisfying users' information demands. More recently, neural rankers fine-tuned
from pre-trained language models (PLMs) establish state-of-the-art ranking
effectiveness. However, it is nontrivial to directly apply these PLM-based
rankers to the large-scale web search system due to the following challenging
issues:(1) the prohibitively expensive computations of massive neural PLMs,
especially for long texts in the web-document, prohibit their deployments in an
online ranking system that demands extremely low latency;(2) the discrepancy
between existing ranking-agnostic pre-training objectives and the ad-hoc
retrieval scenarios that demand comprehensive relevance modeling is another
main barrier for improving the online ranking system;(3) a real-world search
engine typically involves a committee of ranking components, and thus the
compatibility of the individually fine-tuned ranking model is critical for a
cooperative ranking system. In this work, we contribute a series of
successfully applied techniques in tackling these exposed issues when deploying
the state-of-the-art Chinese pre-trained language model, i.e., ERNIE, in the
online search engine system. We first articulate a novel practice to
cost-efficiently summarize the web document and contextualize the resultant
summary content with the query using a cheap yet powerful Pyramid-ERNIE
architecture. Then we endow an innovative paradigm to finely exploit the
large-scale noisy and biased post-click behavioral data for relevance-oriented
pre-training. We also propose a human-anchored fine-tuning strategy tailored
for the online ranking system, aiming to stabilize the ranking signals across
various online components. Extensive offline and online experimental results
show that the proposed techniques significantly boost the search engine's
performance.

本文介绍了将预训练语言模型（PLMs）引入搜索引擎的相关技术及其在实际应用中所面临的问题，使用 ERINE 语言模型以及创新的预训练策略、调整排列信号的方法，成功提高搜索引擎的性能。