The generation of undesirable and factually incorrect content of large
language models poses a significant challenge and remains largely an unsolved
issue. This paper studies the integration of a contrastive learning objective
for fine-tuning LLMs for implicit knowledge editing and controlled text
generation. Optimizing the training objective entails aligning text
perplexities in a contrastive fashion. To facilitate training the model in a
self-supervised fashion, we leverage an off-the-shelf LLM for training data
generation. We showcase applicability in the domain of detoxification. Herein,
the proposed approach leads to a significant decrease in the generation of
toxic content while preserving general utility for downstream tasks such as
commonsense reasoning and reading comprehension. The proposed approach is
conceptually simple but empirically powerful.

研究通过对大型语言模型进行对比学习目标的整合，以实现隐式知识编辑和受控文本生成，从而解决生成不受欢迎和事实不正确的内容的问题。该方法在自毁训练方式的基础上，通过利用现成的语言模型进行数据生成，成功降低了生成有毒内容的频率，并在通用任务（如常识推理和阅读理解）中保持了模型的实用性。该方法简单且实践有效。