Detoxification for LLMs is challenging since it requires models to avoid
generating harmful content while maintaining the generation capability. To
ensure the safety of generations, previous detoxification methods detoxify the
models by changing the data distributions or constraining the generations from
different aspects in a single-step manner. However, these approaches will
dramatically affect the generation quality of LLMs, e.g., discourse coherence
and semantic consistency, since language models tend to generate along the
toxic prompt while detoxification methods work in the opposite direction. To
handle such a conflict, we decompose the detoxification process into different
sub-steps, where the detoxification is concentrated in the input stage and the
subsequent continual generation is based on the non-toxic prompt. Besides, we
also calibrate the strong reasoning ability of LLMs by designing a Detox-Chain
to connect the above sub-steps in an orderly manner, which allows LLMs to
detoxify the text step-by-step. Automatic and human evaluation on two
benchmarks reveals that by training with Detox-Chain, six LLMs scaling from 1B
to 33B can obtain significant detoxification and generation improvement. Our
code and data are available at this https URL Warning:
examples in the paper may contain uncensored offensive content.

针对语言模型的解毒具有挑战性，本文提出了分解解毒过程、基于无毒提示的连续生成以及使用 Detox-Chain 进行有序连接的方法来实现显著的解毒和生成改进。