Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. To address this significant shortcoming, we present RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. Experimental results on the TREC 2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter model, although our effectiveness remains slightly behind reranking with GPT-4. We hope our work provides the foundation for future research on reranking with modern LLMs. All the code necessary to reproduce our results is available at https://github.com/castorini/rank_llm.

我们介绍了RankVicuna，它是第一个完全开源的大型语言模型，能够在零样本情况下进行高质量的列表排序。在TREC 2019和2020深度学习跟踪的实验结果显示，我们可以使用比GPT-3.5小得多的7B参数模型达到与零样本重新排序相当的有效性，尽管我们的有效性略低于GPT-4重新排序。我们希望我们的工作为未来关于使用现代大型语言模型进行重新排序的研究提供基础。我们的全部代码以供重现结果，请访问此https URL。

RankVicuna：零样本列表式文档重排序与开源大型语言模型