Aligning Large Language Models (LLMs) with human values and preferences is
essential for making them helpful and safe. However, building efficient tools
to perform alignment can be challenging, especially for the largest and most
competent LLMs which often contain tens or hundreds of billions of parameters.
We create NeMo-Aligner, a toolkit for model alignment that can efficiently
scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
optimized and scalable implementations for major paradigms of model alignment
such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
our toolkit supports running most of the alignment techniques in a Parameter
Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
extensibility, allowing support for other alignment techniques with minimal
effort. It is open-sourced with Apache 2.0 License and we invite community
contributions at this https URL

用于大规模语言模型对齐的 NeMo-Aligner 工具包可以高效地扩展到使用数百个 GPU 进行训练，支持 Reinforcement Learning from Human Feedback (RLHF)、Direct Preference Optimization (DPO)、SteerLM、Self-Play Fine-Tuning (SPIN) 等主要模型对齐范式，并具有 Parameter Efficient Fine-Tuning (PEFT) 支持。