Large Language Models (LLMs) exhibit impressive capabilities but also present
risks such as biased content generation and privacy issues. One of the current
alignment techniques includes principle-driven integration, but it faces
challenges arising from the imprecision of manually crafted rules and
inadequate risk perception in models without safety training. To address these,
we introduce Guide-Align, a two-stage approach. Initially, a safety-trained
model identifies potential risks and formulates specific guidelines for various
inputs, thereby establishing a comprehensive library of guidelines and models
for input-guidelines retrieval. Subsequently, the retrieval model correlates
new inputs with pertinent guidelines, guiding LLMs in response generation to
ensure safe and high-quality outputs, thus aligning with human values. An
additional optional stage involves fine-tuning a model with new well-aligned
datasets generated through the process implemented in the second stage. Our
method customizes guidelines to accommodate diverse inputs, thereby enhancing
the fine-grainedness and comprehensiveness of the guideline library.
Furthermore, it incorporates safety expertise from a safety-trained LLM through
a lightweight retrieval model. We evaluated our approach on three benchmarks,
demonstrating significant improvements in LLM security and quality. Notably,
our fine-tuned model, Labrador, even at 13 billion parameters, outperforms
GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.

为了解决大型语言模型存在的偏见内容生成和隐私问题，研究引入了 Guide-Align 方法，该方法通过安全训练模型识别潜在风险并建立指南和模型库，然后用于新输入的相关指导，以确保安全和高质量输出，还可以通过精调模型来优化对多样输入的适应性和对指南库的全面性。实验证明该方法显著提高了大型语言模型的安全性和质量，尤其是在 13 亿参数下，优于 GPT-3.5-turbo 和 GPT-4 的对齐能力。