This work presents a novel approach called oracle-checker scheme for
evaluating the answer given by a generative large language model (LLM). Two
types of checkers are presented. The first type of checker follows the idea of
property testing. The second type of checker follows the idea of program
checking. Their applications are demonstrated in two separate contexts, entity
extraction and paraphrase decision, respectively.

本文提出了一种名为 “Oracle-Checker 方案” 的新方法，用于评估生成式大型语言模型（LLM）给出的答案。介绍了两种类型的检查器，一种是基于属性测试思想，另一种是基于程序检查思想。分别展示了它们在实体抽取和近义词判断两个不同背景下的应用。

评估生成式大型语言模型的 Oracle-Checker 方案

Oracle-Checker Scheme for Evaluating a Generative Large Language Model

Objective To solve major clinical natural language processing (NLP) tasks
using a unified text-to-text learning architecture based on a generative large
language model (LLM) via prompt tuning. Methods We formulated 7 key clinical
NLP tasks as text-to-text learning and solved them using one unified generative
clinical LLM, GatorTronGPT, developed using GPT-3 architecture and trained with
up to 20 billion parameters. We adopted soft prompts (i.e., trainable vectors)
with frozen LLM, where the LLM parameters were not updated (i.e., frozen) and
only the vectors of soft prompts were updated, known as prompt tuning. We added
additional soft prompts as a prefix to the input layer, which were optimized
during the prompt tuning. We evaluated the proposed method using 7 clinical NLP
tasks and compared them with previous task-specific solutions based on
Transformer models. Results and Conclusion The proposed approach achieved
state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one
unified generative LLM. Our approach outperformed previous task-specific
transformer models by ~3% for concept extraction and 7% for relation extraction
applied to social determinants of health, 3.4% for clinical concept
normalization, 3.4~10% for clinical abbreviation disambiguation, and 5.5~9% for
natural language inference. Our approach also outperformed a previously
developed prompt-based machine reading comprehension (MRC) model,
GatorTron-MRC, for clinical concept and relation extraction. The proposed
approach can deliver the ``one model for all`` promise from training to
deployment using a unified generative LLM.

通过基于生成型大型语言模型（LLM）的通用文本到文本学习架构和提示调优，解决主要的临床自然语言处理（NLP）任务，并提供了最新的性能。

生成式大型语言模型是全能文本分析引擎：文本对文本学习是您所需的全部

Generative Large Language Models Are All-purpose Text Analytics Engines:  Text-to-text Learning Is All Your Need

Among all the sub-sections in a typical radiology report, the Clinical
Indications, Findings, and Impression often reflect important details about the
health status of a patient. The information included in Impression is also
often covered in Findings. While Findings and Impression can be deduced by
inspecting the image, Clinical Indications often require additional context.
The cognitive task of interpreting medical images remains the most critical and
often time-consuming step in the radiology workflow. Instead of generating an
end-to-end radiology report, in this paper, we focus on generating the Findings
from automated interpretation of medical images, specifically chest X-rays
(CXRs). Thus, this work focuses on reducing the workload of radiologists who
spend most of their time either writing or narrating the Findings. Unlike past
research, which addresses radiology report generation as a single-step image
captioning task, we have further taken into consideration the complexity of
interpreting CXR images and propose a two-step approach: (a) detecting the
regions with abnormalities in the image, and (b) generating relevant text for
regions with abnormalities by employing a generative large language model
(LLM). This two-step approach introduces a layer of interpretability and aligns
the framework with the systematic reasoning that radiologists use when
reviewing a CXR.

本研究提出了一种基于自动医学图像解释的生成式语言模型的方法，用于从胸部 X 线图像中检测异常区域并生成相应文本，以降低放射科医生在撰写或叙述发现结果方面的工作量。