The SuperCLUE-Fin (SC-Fin) benchmark is a pioneering evaluation framework
tailored for Chinese-native financial large language models (FLMs). It assesses
FLMs across six financial application domains and twenty-five specialized
tasks, encompassing theoretical knowledge and practical applications such as
compliance, risk management, and investment analysis. Using multi-turn,
open-ended conversations that mimic real-life scenarios, SC-Fin measures models
on a range of criteria, including accurate financial understanding, logical
reasoning, clarity, computational efficiency, business acumen, risk perception,
and compliance with Chinese regulations.
In a rigorous evaluation involving over a thousand questions, SC-Fin
identifies a performance hierarchy where domestic models like GLM-4 and
MoonShot-v1-128k outperform others with an A-grade, highlighting the potential
for further development in transforming theoretical knowledge into pragmatic
financial solutions. This benchmark serves as a critical tool for refining FLMs
in the Chinese context, directing improvements in financial knowledge
databases, standardizing financial interpretations, and promoting models that
prioritize compliance, risk management, and secure practices.
We create a contextually relevant and comprehensive benchmark that drives the
development of AI in the Chinese financial sector. SC-Fin facilitates the
advancement and responsible deployment of FLMs, offering valuable insights for
enhancing model performance and usability for both individual and institutional
users in the Chinese market..~\footnote{Our benchmark can be found at
https://www.CLUEbenchmarks.com}.

SuperCLUE-Fin 是专为中文金融大型语言模型（FLMs）设计的评估框架，通过六个金融应用领域和 25 个专业任务对 FLMs 进行评估，包括合规性，风险管理和投资分析等理论知识和实际应用。SC-Fin 可帮助改善 FLMs 在中国金融领域的性能和可用性，推动人工智能在这一领域的发展和负责任的应用。

超级 CLUE-Fin：对中文金融 LLM 进行多样化金融任务和应用的分级细粒度分析

SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse  Financial Tasks and Applications

Foundational Language Models (FLMs) have advanced natural language processing
(NLP) research. Current researchers are developing larger FLMs (e.g., XLNet,
T5) to enable contextualized language representation, classification, and
generation. While developing larger FLMs has been of significant advantage, it
is also a liability concerning hallucination and predictive uncertainty.
Fundamentally, larger FLMs are built on the same foundations as smaller FLMs
(e.g., BERT); hence, one must recognize the potential of smaller FLMs which can
be realized through an ensemble. In the current research, we perform a reality
check on FLMs and their ensemble on benchmark and real-world datasets. We
hypothesize that the ensembling of FLMs can influence the individualistic
attention of FLMs and unravel the strength of coordination and cooperation of
different FLMs. We utilize BERT and define three other ensemble techniques:
{Shallow, Semi, and Deep}, wherein the Deep-Ensemble introduces a
knowledge-guided reinforcement learning approach. We discovered that the
suggested Deep-Ensemble BERT outperforms its large variation i.e. BERTlarge, by
a factor of many times using datasets that show the usefulness of NLP in
sensitive fields, such as mental health.

基于大型 FLMs 的小型 FLMs 的潜力是基于同一基础的，通过集成技术可以影响 FLMs 的关注点，揭示不同 FLMs 的协作与合作的能力，我们发现深度集成的 BERT 通过使用心理健康等敏感领域的数据，相对于其大型版本 BERTlarge，具有倍数倍的性能优势。