Modern Large Language Models (LLMs) are capable of following long and complex
instructions that enable a diverse amount of user tasks. However, despite
Information Retrieval (IR) models using LLMs as the backbone of their
architectures, nearly all of them still only take queries as input, with no
instructions. For the handful of recent models that do take instructions, it's
unclear how they use them. We introduce our dataset FollowIR, which contains a
rigorous instruction evaluation benchmark as well as a training set for helping
IR models learn to better follow real-world instructions. FollowIR builds off
the long history of the TREC conferences: as TREC provides human annotators
with instructions (also known as narratives) to determine document relevance,
so should IR models be able to understand and decide relevance based on these
detailed instructions. Our evaluation benchmark starts with three deeply judged
TREC collections and alters the annotator instructions, re-annotating relevant
documents. Through this process, we can measure how well IR models follow
instructions, through a new pairwise evaluation framework. Our results indicate
that existing retrieval models fail to correctly use instructions, using them
for basic keywords and struggling to understand long-form information. However,
we show that it is possible for IR models to learn to follow complex
instructions: our new FollowIR-7B model has significant improvements (over 13%)
after fine-tuning on our training set.

现代大型语言模型具备跟随长而复杂的指令以完成各种用户任务的能力。本研究引入了数据集 FollowIR，其中包含了严格的指令评估基准以及用于帮助信息检索模型学习更好地遵循真实世界指令的训练集。我们的结果表明现有的检索模型未能正确地使用指令，只是简单地使用关键词，并且难以理解长篇信息。然而，我们展示了信息检索模型学习遵循复杂指令的可能性：我们的新模型 FollowIR-7B 在我们的训练集上经过微调后有显著的改进（超过 13%）。