Deployable Large Language Models (LLMs) must conform to the criterion of
helpfulness and harmlessness, thereby achieving consistency between LLMs
outputs and human values. Red-teaming techniques constitute a critical way
towards this criterion. Existing work rely solely on manual red team designs
and heuristic adversarial prompts for vulnerability detection and optimization.
These approaches lack rigorous mathematical formulation, thus limiting the
exploration of diverse attack strategy within quantifiable measure and
optimization of LLMs under convergence guarantees. In this paper, we present
Red-teaming Game (RTG), a general game-theoretic framework without manual
annotation. RTG is designed for analyzing the multi-turn attack and defense
interactions between Red-team language Models (RLMs) and Blue-team Language
Model (BLM). Within the RTG, we propose Gamified Red-teaming Solver (GRTS) with
diversity measure of the semantic space. GRTS is an automated red teaming
technique to solve RTG towards Nash equilibrium through meta-game analysis,
which corresponds to the theoretically guaranteed optimization direction of
both RLMs and BLM. Empirical results in multi-turn attacks with RLMs show that
GRTS autonomously discovered diverse attack strategies and effectively improved
security of LLMs, outperforming existing heuristic red-team designs. Overall,
RTG has established a foundational framework for red teaming tasks and
constructed a new scalable oversight technique for alignment.

用于量化 LLMs 的多样化攻击策略和优化方法，并通过构建红队和蓝队语言模型之间的对抗游戏理论基础，提出一种无需人工标注的红队技术，有效提升了大型语言模型的安全性。