As generative artificial intelligence (AI), particularly Large Language
Models (LLMs), continues to permeate healthcare, it remains crucial to
supplement traditional automated evaluations with human expert evaluation.
Understanding and evaluating the generated texts is vital for ensuring safety,
reliability, and effectiveness. However, the cumbersome, time-consuming, and
non-standardized nature of human evaluation presents significant obstacles to
the widespread adoption of LLMs in practice. This study reviews existing
literature on human evaluation methodologies for LLMs within healthcare. We
highlight a notable need for a standardized and consistent human evaluation
approach. Our extensive literature search, adhering to the Preferred Reporting
Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, spans
publications from January 2018 to February 2024. This review provides a
comprehensive overview of the human evaluation approaches used in diverse
healthcare applications.This analysis examines the human evaluation of LLMs
across various medical specialties, addressing factors such as evaluation
dimensions, sample types, and sizes, the selection and recruitment of
evaluators, frameworks and metrics, the evaluation process, and statistical
analysis of the results. Drawing from diverse evaluation strategies highlighted
in these studies, we propose a comprehensive and practical framework for human
evaluation of generative LLMs, named QUEST: Quality of Information,
Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and
Trust and Confidence. This framework aims to improve the reliability,
generalizability, and applicability of human evaluation of generative LLMs in
different healthcare applications by defining clear evaluation dimensions and
offering detailed guidelines.

该研究回顾了健康医疗领域中基于大型语言模型的人工智能生成文本的人工评估方法，并使用 QUEST 框架提出了一个标准化和统一的人工评估方法，旨在提高可靠性和适用性。

医疗领域生成型大型语言模型人工评估的文献综述与框架

A Literature Review and Framework for Human Evaluation of Generative  Large Language Models in Healthcare

Large language models have proliferated across multiple domains in as short
period of time. There is however hesitation in the medical and healthcare
domain towards their adoption because of issues like factuality, coherence, and
hallucinations. Give the high stakes nature of healthcare, many researchers
have even cautioned against its usage until these issues are resolved. The key
to the implementation and deployment of LLMs in healthcare is to make these
models trustworthy, transparent (as much possible) and explainable. In this
paper we describe the key elements in creating reliable, trustworthy, and
unbiased models as a necessary condition for their adoption in healthcare.
Specifically we focus on the quantification, validation, and mitigation of
hallucinations in the context in healthcare. Lastly, we discuss how the future
of LLMs in healthcare may look like.

在医疗保健领域，大型语言模型的采用存在对可信度、连贯性和幻觉等问题的犹豫。本文讨论了在医疗保健领域中实现可靠、可信赖和无偏倚模型的关键因素，并重点关注在医疗保健背景下幻觉的量化、验证和缓解。最后，我们讨论了大型语言模型在医疗保健中的未来。