Large vision-language models (VLMs) are widely getting adopted in industry
and academia. In this work we build a unified framework to systematically
evaluate gender-profession bias in VLMs. Our evaluation encompasses all
supported inference modes of the recent VLMs, including image-to-text,
text-to-text, text-to-image, and image-to-image. We construct a synthetic,
high-quality dataset of text and images that blurs gender distinctions across
professional actions to benchmark gender bias. In our benchmarking of recent
vision-language models (VLMs), we observe that different input-output
modalities result in distinct bias magnitudes and directions. We hope our work
will help guide future progress in improving VLMs to learn socially unbiased
representations. We will release our data and code.

通过构建统一的框架系统评估大规模视觉语言模型中的性别职业偏见，我们在不同输入输出模态下的基准测试中观察到不同的偏见程度和方向，希望我们的工作能指导未来改进视觉语言模型以学习社会上没有偏见的表示方式。