With an ever-increasing reliance on machine learning (ML) models in the real
world, adversarial examples threaten the safety of AI-based systems such as
autonomous vehicles. In the image domain, they represent maliciously perturbed
data points that look benign to humans (i.e., the image modification is not
noticeable) but greatly mislead state-of-the-art ML models. Previously,
researchers ensured the imperceptibility of their altered data points by
restricting perturbations via $\ell_p$ norms. However, recent publications
claim that creating natural-looking adversarial examples without such
restrictions is also possible. With much more freedom to instill malicious
information into data, these unrestricted adversarial examples can potentially
overcome traditional defense strategies as they are not constrained by the
limitations or patterns these defenses typically recognize and mitigate. This
allows attackers to operate outside of expected threat models. However,
surveying existing image-based methods, we noticed a need for more human
evaluations of the proposed image modifications. Based on existing
human-assessment frameworks for image generation quality, we propose SCOOTER -
an evaluation framework for unrestricted image-based attacks. It provides
researchers with guidelines for conducting statistically significant human
experiments, standardized questions, and a ready-to-use implementation. We
propose a framework that allows researchers to analyze how imperceptible their
unrestricted attacks truly are.

通过对图像领域的方法进行调查，我们注意到对所提议的图像修改进行人类评估的需求。基于现有的图像生成质量人类评估框架，我们提出了 SCOOTER - 一种用于无限制图像攻击的评估框架。它为研究人员提供了进行具有统计意义的人类实验的指南，标准化问题以及可直接使用的实现。我们提出了一种框架，允许研究人员分析他们的无限制攻击是否真正不可察觉。