Semantic textual similartiy (STS) and information retrieval tasks (IR) tasks
have been the two major avenues to record the progress of embedding models in
the past few years. Under the emerging Retrieval-augmented Generation (RAG)
paradigm, we envision the need to evaluate next-level language understanding
abilities of embedding models, and take a conscious look at the reasoning
abilities stored in them. Addressing this, we pose the question: Can retrievers
solve reasoning problems? By transforming reasoning tasks into retrieval tasks,
we find that without specifically trained for reasoning-level language
understanding, current state-of-the-art retriever models may still be far from
being competent for playing the role of assisting LLMs, especially in
reasoning-intensive tasks. Moreover, albeit trained to be aware of
instructions, instruction-aware IR models are often better off without
instructions in inference time for reasoning tasks, posing an overlooked
retriever-LLM behavioral gap for the research community to align. However,
recent decoder-based embedding models show great promise in narrowing the gap,
highlighting the pathway for embedding models to achieve reasoning-level
language understanding. We also show that, although current off-the-shelf
re-ranker models fail on these tasks, injecting reasoning abilities into them
through fine-tuning still appears easier than doing so to bi-encoders, and we
are able to achieve state-of-the-art performance across all tasks by
fine-tuning a reranking model. We release Reasoning as Retrieval Benchmark
(RAR-b), a holistic suite of tasks and settings to evaluate the reasoning
abilities stored in retriever models. RAR-b is available at
this https URL

通过将推理任务转换为检索任务，我们发现，即使没有针对推理级别的语言理解进行专门训练，目前最先进的检索模型在协助 LLMs 的角色上可能仍然远离胜任，特别是在推理密集型任务中。然而，最近基于解码器的嵌入模型在弥合这一差距方面表现出巨大潜力，突显了嵌入模型实现推理级别语言理解的路径。通过微调重新排序模型将推理能力注入其中似乎比对双编码器进行类似操作更容易，我们能够在所有任务中通过微调重新排序模型实现最先进的性能。我们发布由整体任务和设置组成的推理作为检索基准（RAR-b），用于评估检索模型中存储的推理能力。