Vision-Language Pre-training (VLP) models have achieved state-of-the-art
performance in numerous cross-modal tasks. Since they are optimized to capture
the statistical properties of intra- and inter-modality, there remains risk to
learn social biases presented in the data as well. In this work, we (1)
introduce a counterfactual-based bias measurement \emph{CounterBias} to
quantify the social bias in VLP models by comparing the [MASK]ed prediction
probabilities of factual and counterfactual samples; (2) construct a novel
VL-Bias dataset including 24K image-text pairs for measuring gender bias in VLP
models, from which we observed that significant gender bias is prevalent in VLP
models; and (3) propose a VLP debiasing method \emph{FairVLP} to minimize the
difference in the [MASK]ed prediction probabilities between factual and
counterfactual image-text pairs for VLP debiasing. Although CounterBias and
FairVLP focus on social bias, they are generalizable to serve as tools and
provide new insights to probe and regularize more knowledge in VLP models.

本研究针对 Vision-Language Pre-training（VLP）模型的社会偏见问题，提出了一种基于反事实的偏见测量方法 CounterBias、构建了一个包含 24K 图像 - 文本对的新颖 VL-Bias 数据集并在其中观察到了 VLP 模型中普遍存在的显著性别偏见，提出了最小化 VLP 去偏差的思路 FairVLP。