The most recent pointwise Large Language Model (LLM) rankers have achieved
remarkable ranking results. However, these rankers are hindered by two major
drawbacks: (1) they fail to follow a standardized comparison guidance during
the ranking process, and (2) they struggle with comprehensive considerations
when dealing with complicated passages. To address these shortcomings, we
propose to build a ranker that generates ranking scores based on a set of
criteria from various perspectives. These criteria are intended to direct each
perspective in providing a distinct yet synergistic evaluation. Our research,
which examines eight datasets from the BEIR benchmark demonstrates that
incorporating this multi-perspective criteria ensemble approach markedly
enhanced the performance of pointwise LLM rankers.

通过多角度评估的一组标准生成排名得分的排序器显著提高了点对点大型语言模型排名器的性能。

动态生成多样性准则以改善逐点 LLM 排名器

Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers

Large transformer-based language models have been shown to be very effective
in many classification tasks. However, their computational complexity prevents
their use in applications requiring the classification of a large set of
candidates. While previous works have investigated approaches to reduce model
size, relatively little attention has been paid to techniques to improve batch
throughput during inference. In this paper, we introduce the Cascade
Transformer, a simple yet effective technique to adapt transformer-based models
into a cascade of rankers. Each ranker is used to prune a subset of candidates
in a batch, thus dramatically increasing throughput at inference time. Partial
encodings from the transformer model are shared among rerankers, providing
further speed-up. When compared to a state-of-the-art transformer model, our
approach reduces computation by 37% with almost no impact on accuracy, as
measured on two English Question Answering datasets.

本文介绍了级联变压器模型的简单而有效技术，其中通过一系列排名器来修剪批处理中的一部分候选项，从而在推理时显着提高吞吐量，与现有模型相比，我们的方法在不影响准确性的情况下将计算量减少了 37％，并在两个英文问答数据集上进行了评估。