In this paper, we tackle the emerging challenge of unintended harmful content
generation in Large Language Models (LLMs) with a novel dual-stage optimisation
technique using adversarial fine-tuning. Our two-pronged approach employs an
adversarial model, fine-tuned to generate potentially harmful prompts, and a
judge model, iteratively optimised to discern these prompts. In this
adversarial cycle, the two models seek to outperform each other in the
prompting phase, generating a dataset of rich examples which are then used for
fine-tuning. This iterative application of prompting and fine-tuning allows
continuous refinement and improved performance. The performance of our approach
is evaluated through classification accuracy on a dataset consisting of
problematic prompts not detected by GPT-4, as well as a selection of
contentious but unproblematic prompts. We show considerable increase in
classification accuracy of the judge model on this challenging dataset as it
undergoes the optimisation process. Furthermore, we show that a rudimentary
model \texttt{ada} can achieve 13\% higher accuracy on the hold-out test set
than GPT-4 after only a few rounds of this process, and that this fine-tuning
improves performance in parallel tasks such as toxic comment identification.

使用对抗微调的新型双阶段优化技术解决大型语言模型中意外有害内容生成的挑战，并通过分类准确性评估方法展示了优化过程中，判断模型在具有挑战性的数据集上的性能提升。