Natural Language Processing (NLP) is witnessing a remarkable breakthrough
driven by the success of Large Language Models (LLMs). LLMs have gained
significant attention across academia and industry for their versatile
applications in text generation, question answering, and text summarization. As
the landscape of NLP evolves with an increasing number of domain-specific LLMs
employing diverse techniques and trained on various corpus, evaluating
performance of these models becomes paramount. To quantify the performance,
it's crucial to have a comprehensive grasp of existing metrics. Among the
evaluation, metrics which quantifying the performance of LLMs play a pivotal
role. This paper offers a comprehensive exploration of LLM evaluation from a
metrics perspective, providing insights into the selection and interpretation
of metrics currently in use. Our main goal is to elucidate their mathematical
formulations and statistical interpretations. We shed light on the application
of these metrics using recent Biomedical LLMs. Additionally, we offer a
succinct comparison of these metrics, aiding researchers in selecting
appropriate metrics for diverse tasks. The overarching goal is to furnish
researchers with a pragmatic guide for effective LLM evaluation and metric
selection, thereby advancing the understanding and application of these large
language models.

NLP 中，大型语言模型（LLMs）的成功推动了其显著突破，本文对 LLM 的评估方法进行了全面探索，提供了选择和解读已使用度量标准的见解，并采用最新的生物医学 LLM 进行了这些度量标准的应用比较，旨在为研究人员提供一个实用的指南，推进对这些大型语言模型的理解和应用。

揭示以度量为重点的 LLM 评估：挑战与解决方案

Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions

Large Language Models (LLMs) have shown the potential to significantly
contribute to patient care, diagnostics, and administrative processes. Emerging
biomedical LLMs address healthcare-specific challenges, including privacy
demands and computational constraints. However, evaluation of these models has
primarily been limited to non-clinical tasks, which do not reflect the
complexity of practical clinical applications. Additionally, there has been no
thorough comparison between biomedical and general-domain LLMs for clinical
tasks. To fill this gap, we present the Clinical Language Understanding
Evaluation (CLUE), a benchmark tailored to evaluate LLMs on real-world clinical
tasks. CLUE includes two novel datasets derived from MIMIC IV discharge letters
and four existing tasks designed to test the practical applicability of LLMs in
healthcare settings. Our evaluation covers several biomedical and general
domain LLMs, providing insights into their clinical performance and
applicability. CLUE represents a step towards a standardized approach to
evaluating and developing LLMs in healthcare to align future model development
with the real-world needs of clinical application. We publish our evaluation
and data generation scripts: this https URL

为填补现有研究中缺乏对医疗领域广泛应用的临床任务的评估的空白，我们提出了一种适用于现实世界临床任务的基准测试工具 CLUE，并通过评估多个生物医学和通用领域 LLMs 的临床表现和适用性，推进医疗领域的 LLMs 评估和开发的标准化方法。

CLUE: 用于 LLMs 的临床语言理解评估

CLUE: A Clinical Language Understanding Evaluation for LLMs

Medical systematic reviews are crucial for informing clinical decision making
and healthcare policy. But producing such reviews is onerous and
time-consuming. Thus, high-quality evidence synopses are not available for many
questions and may be outdated even when they are available. Large language
models (LLMs) are now capable of generating long-form texts, suggesting the
tantalizing possibility of automatically generating literature reviews on
demand. However, LLMs sometimes generate inaccurate (and potentially
misleading) texts by hallucinating or omitting important information. In the
healthcare context, this may render LLMs unusable at best and dangerous at
worst. Most discussion surrounding the benefits and risks of LLMs have been
divorced from specific applications. In this work, we seek to qualitatively
characterize the potential utility and risks of LLMs for assisting in
production of medical evidence reviews. We conducted 16 semi-structured
interviews with international experts in systematic reviews, grounding
discussion in the context of generating evidence reviews. Domain experts
indicated that LLMs could aid writing reviews, as a tool for drafting or
creating plain language summaries, generating templates or suggestions,
distilling information, crosschecking, and synthesizing or interpreting text
inputs. But they also identified issues with model outputs and expressed
concerns about potential downstream harms of confidently composed but
inaccurate LLM outputs which might mislead. Other anticipated potential
downstream harms included lessened accountability and proliferation of
automatically generated reviews that might be of low quality. Informed by this
qualitative analysis, we identify criteria for rigorous evaluation of
biomedical LLMs aligned with domain expert views.

该研究考虑了使用大型自然语言处理模型辅助生成医学证据综述的益处和风险，以 16 个半结构化面谈为基础，为领域讨论提出了评估标准。