With the rise of AI systems in real-world applications comes the need for
reliable and trustworthy AI. An essential aspect of this are explainable AI
systems. However, there is no agreed standard on how explainable AI systems
should be assessed. Inspired by the Turing test, we introduce a human-centric
assessment framework where a leading domain expert accepts or rejects the
solutions of an AI system and another domain expert. By comparing the
acceptance rates of provided solutions, we can assess how the AI system
performs compared to the domain expert, and whether the AI system's
explanations (if provided) are human-understandable. This setup -- comparable
to the Turing test -- can serve as a framework for a wide range of
human-centric AI system assessments. We demonstrate this by presenting two
instantiations: (1) an assessment that measures the classification accuracy of
a system with the option to incorporate label uncertainties; (2) an assessment
where the usefulness of provided explanations is determined in a human-centric
manner.

本文提出了一个人类中心的评估框架，用于检测可靠的和值得信赖的 AI 系统，以及评估 AI 系统表现如何与领域专家进行比较。该框架可以作为一种人类中心的 AI 系统评估方法。