Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.

本研究解决了当前长上下文语言模型在多跳推理中表现不足的问题，尤其是在嘈杂上下文中。我们提出了“推理与归因”的新方法，通过促使模型在推理过程中为每个断言提供归因，从而增强推理能力。实验结果表明，经过微调的模型在多跳推理基准测试上取得了与现有先进模型接近的竞争性表现。

提升长上下文语言模型的多跳推理能力