Large language models (LLMs), like ChatGPT and GPT-4, have demonstrated
remarkable abilities in natural language understanding and generation. However,
alongside their positive impact on our daily tasks, they can also produce
harmful content that negatively affects societal perceptions. To systematically
assess the safety of Chinese LLMs, we introduce SuperCLUE-Safety (SC-Safety) -
a multi-round adversarial benchmark with 4912 open-ended questions covering
more than 20 safety sub-dimensions. Adversarial human-model interactions and
conversations significantly increase the challenges compared to existing
methods. Experiments on 13 major LLMs supporting Chinese yield the following
insights: 1) Closed-source models outperform open-sourced ones in terms of
safety; 2) Models released from China demonstrate comparable safety levels to
LLMs like GPT-3.5-turbo; 3) Some smaller models with 6B-13B parameters can
compete effectively in terms of safety. By introducing SC-Safety, we aim to
promote collaborative efforts to create safer and more trustworthy LLMs. The
benchmark and findings provide guidance on model selection. Our benchmark can
be found at this https URL

引入了一个多轮对抗基准 (SuperCLUE-Safety)，通过人机对抗交互和对话的方式，系统评估了中文大型语言模型 (Large language models) 的安全性，发现闭源模型在安全性方面优于开源模型，中国发布的模型与 GPT-3.5-turbo 等模型具有可比较的安全水平，某些参数为 6B-13B 的较小模型在安全性方面同样具有有效竞争力，这项基准和发现为模型选择提供了指导。