Citation practices are crucial in shaping the structure of scientific
knowledge, yet they are often influenced by contemporary norms and biases. The
emergence of Large Language Models (LLMs) like GPT-4 introduces a new dynamic
to these practices. Interestingly, the characteristics and potential biases of
references recommended by LLMs that entirely rely on their parametric
knowledge, and not on search or retrieval-augmented generation, remain
unexplored. Here, we analyze these characteristics in an experiment using a
dataset of 166 papers from AAAI, NeurIPS, ICML, and ICLR, published after
GPT-4's knowledge cut-off date, encompassing 3,066 references in total. In our
experiment, GPT-4 was tasked with suggesting scholarly references for the
anonymized in-text citations within these papers. Our findings reveal a
remarkable similarity between human and LLM citation patterns, but with a more
pronounced high citation bias in GPT-4, which persists even after controlling
for publication year, title length, number of authors, and venue. Additionally,
we observe a large consistency between the characteristics of GPT-4's existing
and non-existent generated references, indicating the model's internalization
of citation patterns. By analyzing citation graphs, we show that the references
recommended by GPT-4 are embedded in the relevant citation context, suggesting
an even deeper conceptual internalization of the citation networks. While LLMs
can aid in citation generation, they may also amplify existing biases and
introduce new ones, potentially skewing scientific knowledge dissemination. Our
results underscore the need for identifying the model's biases and for
developing balanced methods to interact with LLMs in general.

大型语言模型（LLMs）如 GPT-4 在引用实践中引入新的动态，并且可能放大现有的偏见和引入新的偏见，从而可能扭曲科学知识的传播。