Recently, many works have proposed various financial large language models
(FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on
financial corpora. However, existing FinLLMs exhibit unsatisfactory performance
in understanding financial text when numeric variables are involved in
questions. In this paper, we propose a novel LLM, called numeric-sensitive
large language model (NumLLM), for Chinese finance. We first construct a
financial corpus from financial textbooks which is essential for improving
numeric capability of LLMs during fine-tuning. After that, we train two
individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed
financial corpus. One module is for adapting general-purpose LLMs to financial
domain, and the other module is for enhancing the ability of NumLLM to
understand financial text with numeric variables. Lastly, we merge the two LoRA
modules into the foundation model to obtain NumLLM for inference. Experiments
on financial question-answering benchmark show that NumLLM can boost the
performance of the foundation model and can achieve the best overall
performance compared to all baselines, on both numeric and non-numeric
questions.

我们提出了一种新颖的大型语言模型（NumLLM），用于改进金融领域的理解能力，该模型通过在构建的金融语料库上进行微调以适应数字变量，并在金融问答基准上取得了最佳性能表现。

NumLLM：针对中国金融领域的数值敏感大型语言模型

NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance

The SuperCLUE-Fin (SC-Fin) benchmark is a pioneering evaluation framework
tailored for Chinese-native financial large language models (FLMs). It assesses
FLMs across six financial application domains and twenty-five specialized
tasks, encompassing theoretical knowledge and practical applications such as
compliance, risk management, and investment analysis. Using multi-turn,
open-ended conversations that mimic real-life scenarios, SC-Fin measures models
on a range of criteria, including accurate financial understanding, logical
reasoning, clarity, computational efficiency, business acumen, risk perception,
and compliance with Chinese regulations.
In a rigorous evaluation involving over a thousand questions, SC-Fin
identifies a performance hierarchy where domestic models like GLM-4 and
MoonShot-v1-128k outperform others with an A-grade, highlighting the potential
for further development in transforming theoretical knowledge into pragmatic
financial solutions. This benchmark serves as a critical tool for refining FLMs
in the Chinese context, directing improvements in financial knowledge
databases, standardizing financial interpretations, and promoting models that
prioritize compliance, risk management, and secure practices.
We create a contextually relevant and comprehensive benchmark that drives the
development of AI in the Chinese financial sector. SC-Fin facilitates the
advancement and responsible deployment of FLMs, offering valuable insights for
enhancing model performance and usability for both individual and institutional
users in the Chinese market..~\footnote{Our benchmark can be found at
https://www.CLUEbenchmarks.com}.

SuperCLUE-Fin 是专为中文金融大型语言模型（FLMs）设计的评估框架，通过六个金融应用领域和 25 个专业任务对 FLMs 进行评估，包括合规性，风险管理和投资分析等理论知识和实际应用。SC-Fin 可帮助改善 FLMs 在中国金融领域的性能和可用性，推动人工智能在这一领域的发展和负责任的应用。