Automatic citation generation for sentences in a document or report is
paramount for intelligence analysts, cybersecurity, news agencies, and
education personnel. In this research, we investigate whether large language
models (LLMs) are capable of generating references based on two forms of
sentence queries: (a) Direct Queries, LLMs are asked to provide author names of
the given research article, and (b) Indirect Queries, LLMs are asked to provide
the title of a mentioned article when given a sentence from a different
article. To demonstrate where LLM stands in this task, we introduce a large
dataset called REASONS comprising abstracts of the 12 most popular domains of
scientific research on arXiv. From around 20K research articles, we make the
following deductions on public and proprietary LLMs: (a) State-of-the-art,
often called anthropomorphic GPT-4 and GPT-3.5, suffers from high pass
percentage (PP) to minimize the hallucination rate (HR). When tested with
Perplexity.ai (7B), they unexpectedly made more errors; (b) Augmenting relevant
metadata lowered the PP and gave the lowest HR; (c) Advance retrieval-augmented
generation (RAG) using Mistral demonstrates consistent and robust citation
support on indirect queries and matched performance to GPT-3.5 and GPT-4. The
HR across all domains and models decreased by an average of 41.93% and the PP
was reduced to 0% in most cases. In terms of generation quality, the average F1
Score and BLEU were 68.09% and 57.51%, respectively; (d) Testing with
adversarial samples showed that LLMs, including the Advance RAG Mistral,
struggle to understand context, but the extent of this issue was small in
Mistral and GPT-4-Preview. Our study con tributes valuable insights into the
reliability of RAG for automated citation generation tasks.

大型语言模型（LLM）是否能根据直接查询和间接查询在文档或报告中生成参考文献是智能分析、网络安全、新闻机构和教育人员至关重要的问题。本研究通过调查大型语言模型在生成参考文献任务中的表现，引入名为 REASONS 的大型数据集，以科学研究领域的 12 个最热门领域的摘要为基础，得出相关结论，并提供有关自动引文生成任务可靠性的有价值的见解。