Fine-tuning (via methods such as instruction-tuning or reinforcement learning
from human feedback) is a crucial step in training language models to robustly
carry out tasks of interest. However, we lack a systematic understanding of the
effects of fine-tuning, particularly on tasks outside the narrow fine-tuning
distribution. In a simplified scenario, we demonstrate that improving
performance on tasks within the fine-tuning data distribution comes at the
expense of suppressing model capabilities on other tasks. This degradation is
especially pronounced for tasks "closest" to the fine-tuning distribution. We
hypothesize that language models implicitly infer the task of the prompt
corresponds, and the fine-tuning process predominantly skews this task
inference towards tasks in the fine-tuning distribution. To test this
hypothesis, we propose Conjugate Prompting to see if we can recover pretrained
capabilities. Conjugate prompting artificially makes the task look farther from
the fine-tuning distribution while requiring the same capability. We find that
conjugate prompting systematically recovers some of the pretraining
capabilities on our synthetic setup. We then apply conjugate prompting to
real-world LLMs using the observation that fine-tuning distributions are
typically heavily skewed towards English. We find that simply translating the
prompts to different languages can cause the fine-tuned models to respond like
their pretrained counterparts instead. This allows us to recover the in-context
learning abilities lost via instruction tuning, and more concerningly, to
recover harmful content generation suppressed by safety fine-tuning in chatbots
like ChatGPT.

微调语言模型会在特定任务上产生性能提升，但对其他任务可能造成能力抑制。为了恢复预训练能力，通过对任务进行翻译使其远离语言模型微调分布，发现可以恢复上下文学习能力，并使聊天机器人生成有害内容。