As the use of deep learning in high impact domains becomes ubiquitous, it is
increasingly important to assess the resilience of models. One such high impact
domain is that of face recognition, with real world applications involving
images affected by various degradations, such as motion blur or high exposure.
Moreover, images captured across different attributes, such as gender and race,
can also challenge the robustness of a face recognition algorithm. While
traditional summary statistics suggest that the aggregate performance of face
recognition models has continued to improve, these metrics do not directly
measure the robustness or fairness of the models. Visual Psychophysics
Sensitivity Analysis (VPSA) [1] provides a way to pinpoint the individual
causes of failure by way of introducing incremental perturbations in the data.
However, perturbations may affect subgroups differently. In this paper, we
propose a new fairness evaluation based on robustness in the form of a generic
framework that extends VPSA. With this framework, we can analyze the ability of
a model to perform fairly for different subgroups of a population affected by
perturbations, and pinpoint the exact failure modes for a subgroup by measuring
targeted robustness. With the increasing focus on the fairness of models, we
use face recognition as an example application of our framework and propose to
compactly visualize the fairness analysis of a model via AUC matrices. We
analyze the performance of common face recognition models and empirically show
that certain subgroups are at a disadvantage when images are perturbed, thereby
uncovering trends that were not visible using the model's performance on
subgroups without perturbations.

本文提出了一种基于 Visual Psychophysics Sensitivity Analysis 扩展的新型公平性评估框架，该框架可分析模型在不同被干扰亚组群中表现得是否公平，并通过 AUC 矩阵来紧凑可视化模型的公平性分析。作者分析了常见人脸识别模型的表现，实证发现当图像受到扰动时，某些亚组群处于不利地位，揭示了在无干扰的子群体上模型表现不明显的趋势。