Existing frameworks for assessing robustness of large language models (LLMs)
overly depend on specific benchmarks, increasing costs and failing to evaluate
performance of LLMs in professional domains due to dataset limitations. This
paper proposes a framework that systematically evaluates the robustness of LLMs
under adversarial attack scenarios by leveraging knowledge graphs (KGs). Our
framework generates original prompts from the triplets of knowledge graphs and
creates adversarial prompts by poisoning, assessing the robustness of LLMs
through the results of these adversarial attacks. We systematically evaluate
the effectiveness of this framework and its modules. Experiments show that
adversarial robustness of the ChatGPT family ranks as GPT-4-turbo > GPT-4o >
GPT-3.5-turbo, and the robustness of large language models is influenced by the
professional domains in which they operate.

本文提出了一种通过知识图谱来评估大型语言模型在对抗攻击场景下的鲁棒性的框架，并通过生成原始提示和毒化的对抗提示来评估模型的鲁棒性。实验证明，ChatGPT 家族的对抗鲁棒性在 GPT-4-turbo > GPT-4o > GPT-3.5-turbo，而大型语言模型的鲁棒性受其所操作的专业领域的影响。