In this paper, we address the limitations of existing text-to-image diffusion
models in generating demographically fair results when given human-related
descriptions. These models often struggle to disentangle the target language
context from sociocultural biases, resulting in biased image generation. To
overcome this challenge, we propose Fair Mapping, a general, model-agnostic,
and lightweight approach that modifies a pre-trained text-to-image model by
controlling the prompt to achieve fair image generation. One key advantage of
our approach is its high efficiency. The training process only requires
updating a small number of parameters in an additional linear mapping network.
This not only reduces the computational cost but also accelerates the
optimization process. We first demonstrate the issue of bias in generated
results caused by language biases in text-guided diffusion models. By
developing a mapping network that projects language embeddings into an unbiased
space, we enable the generation of relatively balanced demographic results
based on a keyword specified in the prompt. With comprehensive experiments on
face image generation, we show that our method significantly improves image
generation performance when prompted with descriptions related to human faces.
By effectively addressing the issue of bias, we produce more fair and diverse
image outputs. This work contributes to the field of text-to-image generation
by enhancing the ability to generate images that accurately reflect the
intended demographic characteristics specified in the text.

通过修改预训练的文本到图像模型以控制提示语，我们提出了一种称之为公平映射的通用、无模型依赖和轻量级方法，以解决现有文本到图像扩散模型在生成人类相关描述时产生有偏见图像的限制。

公平文本到图像扩散通过公平映射

Fair Text-to-Image Diffusion via Fair Mapping

To mitigate the effects of undesired biases in models, several approaches
propose to pre-process the input dataset to reduce the risks of discrimination
by preventing the inference of sensitive attributes. Unfortunately, most of
these pre-processing methods lead to the generation a new distribution that is
very different from the original one, thus often leading to unrealistic data.
As a side effect, this new data distribution implies that existing models need
to be re-trained to be able to make accurate predictions. To address this
issue, we propose a novel pre-processing method, that we coin as fair mapping,
based on the transformation of the distribution of protected groups onto a
chosen target one, with additional privacy constraints whose objective is to
prevent the inference of sensitive attributes. More precisely, we leverage on
the recent works of the Wasserstein GAN and AttGAN frameworks to achieve the
optimal transport of data points coupled with a discriminator enforcing the
protection against attribute inference. Our proposed approach, preserves the
interpretability of data and can be used without defining exactly the sensitive
groups. In addition, our approach can be specialized to model existing
state-of-the-art approaches, thus proposing a unifying view on these methods.
Finally, several experiments on real and synthetic datasets demonstrate that
our approach is able to hide the sensitive attributes, while limiting the
distortion of the data and improving the fairness on subsequent data analysis
tasks.

为了解决模型中存在的错误偏见，本研究提出了一种新的预处理方法 - 公平映射，通过转换敏感组的数据分布到一个特定的目标分布上，并且实现保护敏感属性不被推论的目标来改善数据的公平性。实验结果表明，该方法能够同时隐蔽敏感属性，不失数据的特性，同时提高了后续数据分析任务的公平性。