Adversarial examples are typically optimized with gradient-based attacks.
While novel attacks are continuously proposed, each is shown to outperform its
predecessors using different experimental setups, hyperparameter settings, and
number of forward and backward calls to the target models. This provides
overly-optimistic and even biased evaluations that may unfairly favor one
particular attack over the others. In this work, we aim to overcome these
limitations by proposing AttackBench, i.e., the first evaluation framework that
enables a fair comparison among different attacks. To this end, we first
propose a categorization of gradient-based attacks, identifying their main
components and differences. We then introduce our framework, which evaluates
their effectiveness and efficiency. We measure these characteristics by (i)
defining an optimality metric that quantifies how close an attack is to the
optimal solution, and (ii) limiting the number of forward and backward queries
to the model, such that all attacks are compared within a given maximum query
budget. Our extensive experimental analysis compares more than 100 attack
implementations with a total of over 800 different configurations against
CIFAR-10 and ImageNet models, highlighting that only very few attacks
outperform all the competing approaches. Within this analysis, we shed light on
several implementation issues that prevent many attacks from finding better
solutions or running at all. We release AttackBench as a publicly available
benchmark, aiming to continuously update it to include and evaluate novel
gradient-based attacks for optimizing adversarial examples.

我们提出了 AttackBench 作为第一个用于评估对抗样本优化的攻击的公平比较框架，通过分类梯度攻击、定义优化度量标准和限制查询次数等方式评估攻击的效果和效率，并通过对超过 100 种攻击实现在 CIFAR-10 和 ImageNet 模型上的 800 多种不同配置的实证分析，发现只有很少的攻击能够超越所有其他竞争方法，同时揭示了阻碍许多攻击找到更好解决方案或运行的几个实现问题，我们将 AttackBench 作为一个公开可用的基准，并致力于不断更新以包含和评估新的梯度攻击方法。