The swift progress and widespread acceptance of artificial intelligence (AI)
systems highlight a pressing requirement to comprehend both the capabilities
and potential risks associated with AI. Given the linguistic complexity,
cultural richness, and underrepresented status of Arabic in AI research, there
is a pressing need to focus on Large Language Models (LLMs) performance and
safety for Arabic related tasks. Despite some progress in their development,
there is a lack of comprehensive trustworthiness evaluation benchmarks which
presents a major challenge in accurately assessing and improving the safety of
LLMs when prompted in Arabic. In this paper, we introduce AraTrust 1, the first
comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises
516 human-written multiple-choice questions addressing diverse dimensions
related to truthfulness, ethics, safety, physical health, mental health,
unfairness, illegal activities, privacy, and offensive language. By introducing
AraTrust, we aim to promote collaborative efforts to create safer and more
trustworthy LLMs for Arabic users. We evaluated a set of LLMs against our
benchmark to assess its trustworthiness. GPT-4 showed to be the most
trustworthy regarding Arabic language.

提出了第一个全面的阿拉伯语语言模型信任度基准 ——AraTrust，其中包含了多个方面的真实性、伦理、安全性、身体健康、心理健康、不公正、非法活动、隐私和冒犯性语言的 516 道人工多项选择题。通过引入 AraTrust，旨在促进共同努力，为阿拉伯用户创造更安全、更值得信赖的语言模型，其中 GPT-4 在阿拉伯语方面表现出最高的可信度。