In this work, we present Xwin-LM, a comprehensive suite of alignment
methodologies for large language models (LLMs). This suite encompasses several
key techniques, including supervised finetuning (SFT), reward modeling (RM),
rejection sampling finetuning (RS), and direct preference optimization (DPO).
The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
consistent and significant improvements across the pipeline, demonstrating the
strength and scalability of Xwin-LM. The repository
this https URL will be continually updated to foster
community research.

Xwin-LM 是一套适用于大型语言模型（LLM）的综合对齐方法学，包括 SFT、RM、RS 和 DPO 等关键技术，并通过 AlpacaEval 和 MT-bench 的评估证明了其稳定和可扩展性。