Large Language Models (LLMs) have significantly enhanced Information
Retrieval (IR) across various modules, such as reranking. Despite impressive
performance, current zero-shot relevance ranking with LLMs heavily relies on
human prompt engineering. Existing automatic prompt engineering algorithms
primarily focus on language modeling and classification tasks, leaving the
domain of IR, particularly reranking, underexplored. Directly applying current
prompt engineering algorithms to relevance ranking is challenging due to the
integration of query and long passage pairs in the input, where the ranking
complexity surpasses classification tasks. To reduce human effort and unlock
the potential of prompt optimization in reranking, we introduce a novel
automatic prompt engineering algorithm named APEER. APEER iteratively generates
refined prompts through feedback and preference optimization. Extensive
experiments with four LLMs and ten datasets demonstrate the substantial
performance improvement of APEER over existing state-of-the-art (SoTA) manual
prompts. Furthermore, we find that the prompts generated by APEER exhibit
better transferability across diverse tasks and LLMs. Code is available at
this https URL

通过自动提示工程算法 APEER，大型语言模型在信息检索中的零样本相关性排名取得了显著的性能提升，并展现出更好的跨任务和跨语言模型的可迁移性。