We propose an experimental method for measuring bias in face recognition
systems. Existing methods to measure bias depend on benchmark datasets that are
collected in the wild and annotated for protected (e.g., race, gender) and
non-protected (e.g., pose, lighting) attributes. Such observational datasets
only permit correlational conclusions, e.g., "Algorithm A's accuracy is
different on female and male faces in dataset X.". By contrast, experimental
methods manipulate attributes individually and thus permit causal conclusions,
e.g., "Algorithm A's accuracy is affected by gender and skin color."
Our method is based on generating synthetic faces using a neural face
generator, where each attribute of interest is modified independently while
leaving all other attributes constant. Human observers crucially provide the
ground truth on perceptual identity similarity between synthetic image pairs.
We validate our method quantitatively by evaluating race and gender biases of
three research-grade face recognition models. Our synthetic pipeline reveals
that for these algorithms, accuracy is lower for Black and East Asian
population subgroups. Our method can also quantify how perceptual changes in
attributes affect face identity distances reported by these models. Our large
synthetic dataset, consisting of 48,000 synthetic face image pairs (10,200
unique synthetic faces) and 555,000 human annotations (individual attributes
and pairwise identity comparisons) is available to researchers in this
important area.

我们提出了一个实验方法来测量人脸识别系统中的偏见，使用神经人脸生成器生成合成人脸，改变感兴趣的属性而保持其他属性不变，并通过人类观察者提供关于感知身份相似性的真实数据，验证了该方法在种族和性别方面的偏见，同时提供了大规模的合成数据集，可用于相关领域的研究。