The increasing use of foundation models highlights the urgent need to address
and eliminate implicit biases present in them that arise during pretraining. In
this paper, we introduce PEFTDebias, a novel approach that employs
parameter-efficient fine-tuning (PEFT) to mitigate the biases within foundation
models. PEFTDebias consists of two main phases: an upstream phase for acquiring
debiasing parameters along a specific bias axis, and a downstream phase where
these parameters are incorporated into the model and frozen during the
fine-tuning process. By evaluating on four datasets across two bias axes namely
gender and race, we find that downstream biases can be effectively reduced with
PEFTs. In addition, we show that these parameters possess axis-specific
debiasing characteristics, enabling their effective transferability in
mitigating biases in various downstream tasks. To ensure reproducibility, we
release the code to do our experiments.

通过引入 PEFTDebias 方法，在 foundation models 中使用参数高效微调 (PEFT) 来减轻隐含的偏见。PEFTDebias 由两个主要阶段组成：上游阶段用于沿特定偏见轴获取去偏参数，下游阶段将这些参数整合到模型中，在微调过程中固定。通过在两个偏见轴（性别和种族）上的四个数据集上进行评估，我们发现用 PEFT 可以有效地减少下游偏见。此外，我们还展示了这些参数具有特定于轴的去偏特性，可以在各种下游任务中有效地减轻偏见。为了确保可重复性，我们公开了我们的实验代码。