Large language models (LLMs) can generate natural language texts for various
domains and tasks, but their potential for clinical text mining, a domain with
scarce, sensitive, and imbalanced medical data, is underexplored. We
investigate whether LLMs can augment clinical data for detecting Alzheimer's
Disease (AD)-related signs and symptoms from electronic health records (EHRs),
a challenging task that requires high expertise. We create a novel pragmatic
taxonomy for AD sign and symptom progression based on expert knowledge, which
guides LLMs to generate synthetic data following two different directions:
"data-to-label", which labels sentences from a public EHR collection with
AD-related signs and symptoms; and "label-to-data", which generates sentences
with AD-related signs and symptoms based on the label definition. We train a
system to detect AD-related signs and symptoms from EHRs, using three datasets:
(1) a gold dataset annotated by human experts on longitudinal EHRs of AD
patients; (2) a silver dataset created by the data-to-label method; and (3) a
bronze dataset created by the label-to-data method. We find that using the
silver and bronze datasets improves the system performance, outperforming the
system using only the gold dataset. This shows that LLMs can generate synthetic
clinical data for a complex task by incorporating expert knowledge, and our
label-to-data method can produce datasets that are free of sensitive
information, while maintaining acceptable quality.

通过利用大规模语言模型生成合成的临床数据，我们研究了其在检测阿尔茨海默病相关迹象和症状方面的潜力，证明了大规模语言模型能够提高系统性能并生成不泄露敏感信息的数据集。

大型语言模型中的临床数据生成的两个方向：数据到标签和标签到数据

Two Directions for Clinical Data Generation with Large Language Models:  Data-to-Label and Label-to-Data

Recent advancements in large language models (LLMs) have led to the
development of highly potent models like OpenAI's ChatGPT. These models have
exhibited exceptional performance in a variety of tasks, such as question
answering, essay composition, and code generation. However, their effectiveness
in the healthcare sector remains uncertain. In this study, we seek to
investigate the potential of ChatGPT to aid in clinical text mining by
examining its ability to extract structured information from unstructured
healthcare texts, with a focus on biological named entity recognition and
relation extraction. However, our preliminary results indicate that employing
ChatGPT directly for these tasks resulted in poor performance and raised
privacy concerns associated with uploading patients' information to the ChatGPT
API. To overcome these limitations, we propose a new training paradigm that
involves generating a vast quantity of high-quality synthetic data with labels
utilizing ChatGPT and fine-tuning a local model for the downstream task. Our
method has resulted in significant improvements in the performance of
downstream tasks, improving the F1-score from 23.37% to 63.99% for the named
entity recognition task and from 75.86% to 83.59% for the relation extraction
task. Furthermore, generating data using ChatGPT can significantly reduce the
time and effort required for data collection and labeling, as well as mitigate
data privacy concerns. In summary, the proposed framework presents a promising
solution to enhance the applicability of LLM models to clinical text mining.

本研究旨在探究将 ChatGPT 应用于临床文本挖掘中，针对生物命名实体识别和关系抽取，我们提出了一种基于生成大量合成数据进行本地模型微调的训练范式。结果表明，这种方法显著提高了下游任务的性能，同时缓解了数据隐私问题。

低语言模型 (Synthetic Data Generation of LLMs) 是否有助于临床文本挖掘？

Does Synthetic Data Generation of LLMs Help Clinical Text Mining?

Pre-trained language models (PLMs) have been deployed in many natural
language processing (NLP) tasks and in various domains. Language model
pre-training from general or mixed domain rich data plus fine-tuning using
small amounts of available data in a low resource domain demonstrated
beneficial results by researchers. In this work, we question this statement and
verify if BERT-based PLMs from the biomedical domain can perform well in
clinical text mining tasks via fine-tuning. We test the state-of-the-art
models, i.e. Bioformer which is pre-trained on a large amount of biomedical
data from PubMed corpus. We use a historical n2c2 clinical NLP challenge
dataset for fine-tuning its task-adapted version (BioformerApt), and show that
their performances are actually very low. We also present our own end-to-end
model, TransformerCRF, which is developed using Transformer and conditional
random fields (CRFs) as encoder and decoder. We further create a new variation
model by adding a CRF layer on top of PLM Bioformer (BioformerCRF). We
investigate the performances of TransformerCRF on clinical text mining tasks by
training from scratch using a limited amount of data, as well as the model
BioformerCRF. Experimental evaluation shows that, in a \textit{constrained
setting}, all tested models are \textit{far from ideal} regarding extreme
low-frequency special token recognition, even though they can achieve
relatively higher accuracy on overall text tagging. Our models including source
codes will be hosted at https://github.com/poethan/TransformerCRF.

本文通过使用历史数据比较了预处理训练的生物医学领域模型（PLMs）和使用 Transformer 和 CRFs 模型的模型（TransformerCRF），并发现在临床文本挖掘任务中它们的性能表现最差。