There exist both scalable tasks, like reading comprehension and
fact-checking, where model performance improves with model size, and unscalable
tasks, like arithmetic reasoning and symbolic reasoning, where model
performance does not necessarily improve with model size. Large language models
(LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate
incremental predictions even on unscalable tasks. Unfortunately, despite their
exceptional reasoning abilities, LLMs tend to internalize and reproduce
discriminatory societal biases. Whether CoT can provide discriminatory or
egalitarian rationalizations for the implicit information in unscalable tasks
remains an open question.
In this study, we examine the impact of LLMs' step-by-step predictions on
gender bias in unscalable tasks. For this purpose, we construct a benchmark for
an unscalable task where the LLM is given a list of words comprising feminine,
masculine, and gendered occupational words, and is required to count the number
of feminine and masculine words. In our CoT prompts, we require the LLM to
explicitly indicate whether each word in the word list is a feminine or
masculine before making the final predictions. With counting and handling the
meaning of words, this benchmark has characteristics of both arithmetic
reasoning and symbolic reasoning. Experimental results in English show that
without step-by-step prediction, most LLMs make socially biased predictions,
despite the task being as simple as counting words. Interestingly, CoT
prompting reduces this unconscious social bias in LLMs and encourages fair
predictions.

对于大型语言模型，通过链判断（Chain-of-Thought）激励可以在扩展任务上实现准确的增量预测，但其内化和复制社会偏见仍存在问题，因此需要对其在性别偏见上的影响进行研究。通过创建一个包含女性、男性和有性别意味的职业单词列表的非扩展任务用于构建一个评测基准，研究结果表明，大部分语言模型在不考虑链判断的情况下会做出带有社会偏见的预测，而链判断能够减少语言模型中的潜意识社会偏见，并促使公平的预测。