The task of image captioning implicitly involves gender identification.
However, due to the gender bias in data, gender identification by an image
captioning model suffers. Also, the gender-activity bias, owing to the
word-by-word prediction, influences other words in the caption prediction,
resulting in the well-known problem of label bias. In this work, we investigate
gender bias in the COCO captioning dataset and show that it engenders not only
from the statistical distribution of genders with contexts but also from the
flawed annotation by the human annotators. We look at the issues created by
this bias in the trained models. We propose a technique to get rid of the bias
by splitting the task into 2 subtasks: gender-neutral image captioning and
gender classification. By this decoupling, the gender-context influence can be
eradicated. We train the gender-neutral image captioning model, which gives
comparable results to a gendered model even when evaluating against a dataset
that possesses a similar bias as the training data. Interestingly, the
predictions by this model on images with no humans, are also visibly different
from the one trained on gendered captions. We train gender classifiers using
the available bounding box and mask-based annotations for the person in the
image. This allows us to get rid of the context and focus on the person to
predict the gender. By substituting the genders into the gender-neutral
captions, we get the final gendered predictions. Our predictions achieve
similar performance to a model trained with gender, and at the same time are
devoid of gender bias. Finally, our main result is that on an
anti-stereotypical dataset, our model outperforms a popular image captioning
model which is trained with gender.

本文研究了 COCO 图像描述数据集中的性别偏见对训练模型的影响以及如何通过解耦图像描述和性别分类的方法来消除偏见。

揭示和纠正图像字幕数据集和模型中的性别偏见

Exposing and Correcting the Gender Bias in Image Captioning Datasets and  Models

Word embedding models have become a fundamental component in a wide range of
Natural Language Processing (NLP) applications. However, embeddings trained on
human-generated corpora have been demonstrated to inherit strong gender
stereotypes that reflect social constructs. To address this concern, in this
paper, we propose a novel training procedure for learning gender-neutral word
embeddings. Our approach aims to preserve gender information in certain
dimensions of word vectors while compelling other dimensions to be free of
gender influence. Based on the proposed method, we generate a Gender-Neutral
variant of GloVe (GN-GloVe). Quantitative and qualitative experiments
demonstrate that GN-GloVe successfully isolates gender information without
sacrificing the functionality of the embedding model.

本文提出了一种基于新颖训练过程的性别中性词向量的学习方法，通过在某些维度中保留性别信息并让其他维度不受性别影响的方法，成功地建立了一种 GN-GloVe 的性别中性变体，该变体在保留嵌入模型功能的基础上隔离了性别信息，并得到了定量和定性实验证实。