The hallucination issue is recognized as a fundamental deficiency of large
language models (LLMs), especially when applied to fields such as finance,
education, and law. Despite the growing concerns, there has been a lack of
empirical investigation. In this paper, we provide an empirical examination of
LLMs' hallucination behaviors in financial tasks. First, we empirically
investigate LLM model's ability of explaining financial concepts and
terminologies. Second, we assess LLM models' capacity of querying historical
stock prices. Third, to alleviate the hallucination issue, we evaluate the
efficacy of four practical methods, including few-shot learning, Decoding by
Contrasting Layers (DoLa), the Retrieval Augmentation Generation (RAG) method
and the prompt-based tool learning method for a function to generate a query
command. Finally, our major finding is that off-the-shelf LLMs experience
serious hallucination behaviors in financial tasks. Therefore, there is an
urgent need to call for research efforts in mitigating LLMs' hallucination.

现有的大型语言模型在金融任务中存在严重的幻觉问题，迫切需要研究努力来缓解这一问题。

金融领域大型语言模型的不足：幻觉的实证研究

Deficiency of Large Language Models in Finance: An Empirical Examination  of Hallucination

In the context of few-shot learning, it is currently believed that a fixed
pre-trained (PT) model, along with fine-tuning the final layer during
evaluation, outperforms standard meta-learning algorithms. We re-evaluate these
claims under an in-depth empirical examination of an extensive set of formally
diverse datasets and compare PT to Model Agnostic Meta-Learning (MAML). Unlike
previous work, we emphasize a fair comparison by using: the same architecture,
the same optimizer, and all models trained to convergence. Crucially, we use a
more rigorous statistical tool -- the effect size (Cohen's d) -- to determine
the practical significance of the difference between a model trained with PT
vs. a MAML. We then use a previously proposed metric -- the diversity
coefficient -- to compute the average formal diversity of a dataset. Using this
analysis, we demonstrate the following: 1. when the formal diversity of a data
set is low, PT beats MAML on average and 2. when the formal diversity is high,
MAML beats PT on average. The caveat is that the magnitude of the average
difference between a PT vs. MAML using the effect size is low (according to
classical statistical thresholds) -- less than 0.2. Nevertheless, this
observation is contrary to the currently held belief that a pre-trained model
is always better than a meta-learning model. Our extensive experiments consider
21 few-shot learning benchmarks, including the large-scale few-shot learning
dataset Meta-Data set. We also show no significant difference between a MAML
model vs. a PT model with GPT-2 on Openwebtext. We, therefore, conclude that a
pre-trained model does not always beat a meta-learned model and that the formal
diversity of a dataset is a driving factor.

本文通过深入地实证研究多个数据集，比较了在 few-shot learning 背景下进行微调的预训练模型和标准元学习算法，研究结果表明，预训练模型在数据集形式多样性较低时表现优秀，而当形式多样性较高时，元学习算法表现更优。