Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. These disparities have been shown to exist, but until now there has not been a unified approach to measure these differences for common use-cases of computer vision models. We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, we hired expert reviewers to manually annotate person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes such as disk jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art vision models and present a deeper understanding of potential performance disparities and challenges across sensitive demographic attributes. With the exhaustive annotations collected, we probe models using single demographics attributes as well as multiple attributes using an intersectional approach (e.g. hair color and perceived skin tone). Our results show that classification, detection, segmentation, and visual grounding models exhibit performance disparities across demographic attributes and intersections of attributes. These harms suggest that not all people represented in datasets receive fair and equitable treatment in these vision tasks. We hope current and future results using our benchmark will contribute to fairer, more robust vision models. FACET is available publicly at https://facet.metademolab.com/

计算机视觉模型在性别和肤色等属性方面存在已知的性能差异。本论文提出了一个名为FACET（计算机视觉公平性评估）的新基准，它是一个大型、公开可用的评估集，包括32,000张图片，用于图像分类、目标检测和分割等常见视觉任务。我们通过对FACET中的每张图像进行手动标注，包括感知肤色和发型类型等与人相关的属性，手动绘制边界框，并标记诸如唱片骑师或吉他手等精细的与人相关的类别。我们利用FACET对最先进的视觉模型进行基准测试，并展示潜在的性能差异和挑战。我们的研究结果显示，分类、检测、分割和视觉定位模型在人口属性和属性交叉上存在性能差异。这些伤害表明，在这些视觉任务中，并不是所有的人都能得到公平和公正的对待。我们希望使用我们的基准测试结果能够为更加公平、更加稳健的视觉模型做出贡献。FACET在该https URL上公开可用。

FACET：计算机视觉评估基准的公平性