The accurate evaluation of differential treatment in language models to
specific groups is critical to ensuring a positive and safe user experience. An
ideal evaluation should have the properties of being robust, extendable to new
groups or attributes, and being able to capture biases that appear in typical
usage (rather than just extreme, rare cases). Relatedly, bias evaluation should
surface not only egregious biases but also ones that are subtle and
commonplace, such as a likelihood for talking about appearances with regard to
women. We present FairPair, an evaluation framework for assessing differential
treatment that occurs during ordinary usage. FairPair operates through
counterfactual pairs, but crucially, the paired continuations are grounded in
the same demographic group, which ensures equivalent comparison. Additionally,
unlike prior work, our method factors in the inherent variability that comes
from the generation process itself by measuring the sampling variability. We
present an evaluation of several commonly used generative models and a
qualitative analysis that indicates a preference for discussing family and
hobbies with regard to women.

FairPair 是一个评估框架，用于评估在普通使用中发生的差异对待；通过对具有相同人口群体背景的对照组进行分析，考虑了生成过程本身的变异性，表明更倾向于将家庭和爱好与女性相关联。

公正对待：通过成对扰动对语言模型中的偏见进行稳健评估

FairPair: A Robust Evaluation of Biases in Language Models through  Paired Perturbations

We present a novel convolutional neural network (CNN) design for facial
landmark coordinate regression. We examine the intermediate features of a
standard CNN trained for landmark detection and show that features extracted
from later, more specialized layers capture rough landmark locations. This
provides a natural means of applying differential treatment midway through the
network, tweaking processing based on facial alignment. The resulting Tweaked
CNN model (TCNN) harnesses the robustness of CNNs for landmark detection, in an
appearance-sensitive manner without training multi-part or multi-scale models.
Our results on standard face landmark detection and face verification
benchmarks show TCNN to surpasses previously published performances by wide
margins.

我们提出了一种新颖的卷积神经网络，用于面部标志坐标回归。我们调查了训练用于标志检测的标准 CNN 的中间特征，并展示了从更专业的层提取的特征捕获了大致的地标位置。这为在网络中途应用差分处理提供了自然手段，根据面部对齐调整处理。所得的 TCNN 模型，以外观敏感的方式利用 CNN 对标志检测的健壮性，而不用训练多部分或多尺度模型。我们在标准面部标志检测和面部验证基准测试上的结果显示，TCNN 超过了以前发表的表现。