The usage of Artificial Intelligence (AI) systems has increased
exponentially, thanks to their ability to reduce the amount of data to be
analyzed, the user efforts and preserving a high rate of accuracy. However,
introducing this new element in the loop has converted them into attacked
points that can compromise the reliability of the systems. This new scenario
has raised crucial challenges regarding the reliability and trustworthiness of
the AI models, as well as about the uncertainties in their response decisions,
becoming even more crucial when applied in critical domains such as healthcare,
chemical, electrical plants, etc. To contain these issues, in this paper, we
present NeuralSentinel (NS), a tool able to validate the reliability and
trustworthiness of AI models. This tool combines attack and defence strategies
and explainability concepts to stress an AI model and help non-expert staff
increase their confidence in this new system by understanding the model
decisions. NS provide a simple and easy-to-use interface for helping humans in
the loop dealing with all the needed information. This tool was deployed and
used in a Hackathon event to evaluate the reliability of a skin cancer image
detector. During the event, experts and non-experts attacked and defended the
detector, learning which factors were the most important for model
misclassification and which techniques were the most efficient. The event was
also used to detect NS's limitations and gather feedback for further
improvements.

为了解决人工智能模型的可靠性和可信度问题，本文提出了一个名为 NeuralSentinel（NS）的工具，结合攻击和防御策略以及解释性概念来验证 AI 模型的可靠性和可信度，并帮助非专业人员通过理解模型决策来增强他们对这一新系统的信心。该工具在一个黑客马拉松活动中部署和使用，用于评估皮肤癌检测器的可靠性，并通过攻击和防御活动学习出导致模型错误分类的最重要因素和最有效的技术，同时检测 NS 的局限性并收集反馈以进行进一步改进。