Recent researches indicate that Pre-trained Large Language Models (LLMs)
possess cognitive constructs similar to those observed in humans, prompting
researchers to investigate the cognitive aspects of LLMs. This paper focuses on
explicit and implicit social bias, a distinctive two-level cognitive construct
in psychology. It posits that individuals' explicit social bias, which is their
conscious expression of bias in the statements, may differ from their implicit
social bias, which represents their unconscious bias. We propose a two-stage
approach and discover a parallel phenomenon in LLMs known as "re-judge
inconsistency" in social bias. In the initial stage, the LLM is tasked with
automatically completing statements, potentially incorporating implicit social
bias. However, in the subsequent stage, the same LLM re-judges the biased
statement generated by itself but contradicts it. We propose that this re-judge
inconsistency can be similar to the inconsistency between human's unaware
implicit social bias and their aware explicit social bias. Experimental
investigations on ChatGPT and GPT-4 concerning common gender biases examined in
psychology corroborate the highly stable nature of the re-judge inconsistency.
This finding may suggest that diverse cognitive constructs emerge as LLMs'
capabilities strengthen. Consequently, leveraging psychological theories can
provide enhanced insights into the underlying mechanisms governing the
expressions of explicit and implicit constructs in LLMs.

近期的研究表明，预训练的大型语言模型（LLMs）具有类似于人类观察到的认知结构，促使研究人员探究 LLMs 的认知方面。本文聚焦于心理学中的一个独特两级认知结构，即显性社会偏见和隐性社会偏见。我们提出了一个两阶段的方法，并在 LLMs 中发现了一个并行现象，即社会偏见中的 “重新判断不一致”。实验调查 ChatGPT 和 GPT-4 关于心理学中常见的性别偏见，证实了重新判断不一致的高度稳定性。这一发现可能表明，随着 LLMs 的能力增强，不同的认知结构也会出现。因此，利用心理学理论可以提供对 LLMs 中显性和隐性结构表达的潜在机制的深入洞察。