In this work, we present Xwin-LM, a comprehensive suite of alignment
methodologies for large language models (LLMs). This suite encompasses several
key techniques, including supervised finetuning (SFT), reward modeling (RM),
rejection sampling finetuning (RS), and direct preference optimization (DPO).
The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
consistent and significant improvements across the pipeline, demonstrating the
strength and scalability of Xwin-LM. The repository
this https URL will be continually updated to foster
community research.

Xwin-LM 是一套适用于大型语言模型（LLM）的综合对齐方法学，包括 SFT、RM、RS 和 DPO 等关键技术，并通过 AlpacaEval 和 MT-bench 的评估证明了其稳定和可扩展性。

Xwin-LM：强大且可扩展的 LLM 对齐实践

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Recent years have witnessed remarkable progress made in large language models
(LLMs). Such advancements, while garnering significant attention, have
concurrently elicited various concerns. The potential of these models is
undeniably vast; however, they may yield texts that are imprecise, misleading,
or even detrimental. Consequently, it becomes paramount to employ alignment
techniques to ensure these models to exhibit behaviors consistent with human
values.
This survey endeavors to furnish an extensive exploration of alignment
methodologies designed for LLMs, in conjunction with the extant capability
research in this domain. Adopting the lens of AI alignment, we categorize the
prevailing methods and emergent proposals for the alignment of LLMs into outer
and inner alignment. We also probe into salient issues including the models'
interpretability, and potential vulnerabilities to adversarial attacks. To
assess LLM alignment, we present a wide variety of benchmarks and evaluation
methodologies. After discussing the state of alignment research for LLMs, we
finally cast a vision toward the future, contemplating the promising avenues of
research that lie ahead.
Our aspiration for this survey extends beyond merely spurring research
interests in this realm. We also envision bridging the gap between the AI
alignment research community and the researchers engrossed in the capability
exploration of LLMs for both capable and safe LLMs.

大型语言模型（LLMs）的对齐方法研究，包括外部和内部对齐方法，探讨了其可解释性和对抗攻击的潜在漏洞以及评估方法，并展望了未来的研究方向。