Measuring algorithmic bias is crucial both to assess algorithmic fairness,
and to guide the improvement of algorithms. Current methods to measure
algorithmic bias in computer vision, which are based on observational datasets,
are inadequate for this task because they conflate algorithmic bias with
dataset bias.
To address this problem we develop an experimental method for measuring
algorithmic bias of face analysis algorithms, which manipulates directly the
attributes of interest, e.g., gender and skin tone, in order to reveal causal
links between attribute variation and performance change. Our proposed method
is based on generating synthetic ``transects'' of matched sample images that
are designed to differ along specific attributes while leaving other attributes
constant. A crucial aspect of our approach is relying on the perception of
human observers, both to guide manipulations, and to measure algorithmic bias.
Besides allowing the measurement of algorithmic bias, synthetic transects
have other advantages with respect to observational datasets: they sample
attributes more evenly allowing for more straightforward bias analysis on
minority and intersectional groups, they enable prediction of bias in new
scenarios, they greatly reduce ethical and legal challenges, and they are
economical and fast to obtain, helping make bias testing affordable and widely
available.
We validate our method by comparing it to a study that employs the
traditional observational method for analyzing bias in gender classification
algorithms. The two methods reach different conclusions. While the
observational method reports gender and skin color biases, the experimental
method reveals biases due to gender, hair length, age, and facial hair.

我们提出了一种实验方法来测量面部分析算法的算法偏差，该方法可以操作感兴趣的属性（如性别和肤色），以揭示属性变化与性能变化之间的因果联系，并且相较于观察性数据集，合成横断面具有更均衡的属性采样，可预测新场景中的偏差，在减少道德和法律难题的同时，还可以节省成本，并使偏差测试变得更加负担得起和普遍可用。