Large Language Models (LLMs) such as Mistral and LLaMA have showcased
remarkable performance across various natural language processing (NLP) tasks.
Despite their success, these models inherit social biases from the diverse
datasets on which they are trained. This paper investigates the propagation of
biases within LLMs through a novel feature-based analytical approach. Drawing
inspiration from causal mediation analysis, we hypothesize the evolution of
bias-related features and validate them using interpretability techniques like
activation and attribution patching. Our contributions are threefold: (1) We
introduce and empirically validate a feature-based method for bias analysis in
LLMs, applied to LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3 with templates
from a professions dataset. (2) We extend our method to another form of gender
bias, demonstrating its generalizability. (3) We differentiate the roles of
MLPs and attention heads in bias propagation and implement targeted debiasing
using a counterfactual dataset. Our findings reveal the complex nature of bias
in LLMs and emphasize the necessity for tailored debiasing strategies, offering
a deeper understanding of bias mechanisms and pathways for effective
mitigation.

大型语言模型（LLMs）具有卓越的性能，在各种自然语言处理（NLP）任务中表现出色。本文通过一种新颖的基于特征的分析方法研究了 LLMs 内部偏见的传播。我们的发现揭示了 LLMs 中偏见的复杂性质并强调了定制的去偏方法的必要性，为有效缓解偏见机制和途径提供了更深入的理解。