We introduce Eurus, a suite of large language models (LLMs) optimized for
reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
state-of-the-art results among open-source models on a diverse set of
benchmarks covering mathematics, code generation, and logical reasoning
problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
comprehensive benchmarking across 12 tests covering five tasks, and achieves a
33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
benchmarks, substantially outperforming existing open-source models by margins
more than 13.3%. The strong performance of Eurus can be primarily attributed to
UltraInteract, our newly-curated large-scale, high-quality alignment dataset
specifically designed for complex reasoning tasks. UltraInteract can be used in
both supervised fine-tuning and preference learning. For each instruction, it
includes a preference tree consisting of (1) reasoning chains with diverse
planning strategies in a unified format, (2) multi-turn interaction
trajectories with the environment and the critique, and (3) pairwise data to
facilitate preference learning. UltraInteract allows us to conduct an in-depth
exploration of preference learning for reasoning tasks. Our investigation
reveals that some well-established preference learning algorithms may be less
suitable for reasoning tasks compared to their effectiveness in general
conversations. Inspired by this, we derive a novel reward modeling objective
which, together with UltraInteract, leads to a strong reward model.

Eurus 是一套针对推理进行优化的大型语言模型，通过基于 Mistral-7B 和 CodeLlama-70B 的微调，在数学、代码生成和逻辑推理问题的多种基准测试中取得了领先的结果。通过在五个任务上全面进行 12 项测试对比，Eurus-70B 在 LeetCode 上达到了 33.3% 的一次通过准确率，并且在 TheoremQA 上达到了 32.6%，这两个具有挑战性的基准测试远远超过了现有开源模型 13.3% 以上的性能。Eurus 的强大性能主要归功于 UltraInteract，这是我们专门为复杂推理任务设计的大规模、高质量的对齐数据集，可用于监督微调和偏好学习。通过对偏好学习算法进行深入探究，我们发现某些已建立的偏好学习算法在推理任务中的适用性较差，相比其在一般对话中的有效性。受此启发，我们推导出了一种新颖的奖励建模目标，与 UltraInteract 结合使用可获得强大的奖励模型。