Although supervised machine learning is popular for information extraction
from clinical notes, creating large annotated datasets requires extensive
domain expertise and is time-consuming. Meanwhile, large language models (LLMs)
have demonstrated promising transfer learning capability. In this study, we
explored whether recent LLMs can reduce the need for large-scale data
annotations. We curated a manually-labeled dataset of 769 breast cancer
pathology reports, labeled with 13 categories, to compare zero-shot
classification capability of the GPT-4 model and the GPT-3.5 model with
supervised classification performance of three model architectures: random
forests classifier, long short-term memory networks with attention (LSTM-Att),
and the UCSF-BERT model. Across all 13 tasks, the GPT-4 model performed either
significantly better than or as well as the best supervised model, the LSTM-Att
model (average macro F1 score of 0.83 vs. 0.75). On tasks with high imbalance
between labels, the differences were more prominent. Frequent sources of GPT-4
errors included inferences from multiple samples and complex task design. On
complex tasks where large annotated datasets cannot be easily collected, LLMs
can reduce the burden of large-scale data labeling. However, if the use of LLMs
is prohibitive, the use of simpler supervised models with large annotated
datasets can provide comparable results. LLMs demonstrated the potential to
speed up the execution of clinical NLP studies by reducing the need for
curating large annotated datasets. This may result in an increase in the
utilization of NLP-based variables and outcomes in observational clinical
studies.

通过对 769 份乳腺癌病理报告进行人工标注的数据集的研究，我们比较了 GPT-4 模型和 GPT-3.5 模型的零样本分类能力与三种模型架构的监督分类性能，发现 GPT-4 模型在所有 13 个任务中要么显著优于要么与最佳监督模型的 LSTM-Att 模型相当（平均宏 F1 得分为 0.83 vs. 0.75）。研究还表明，如果难以使用大规模标注数据集，LLMs 可以减轻数据标注的负担，但使用带有大规模标注数据集的简单监督模型也可以提供可比较的结果，LLMs 可以加快临床自然语言处理研究的执行速度，提高 NLP 变量和结果在临床观察研究中的利用率。

大型语言模型与监督建模在乳腺癌病理分类中零样本推理的比较研究

A comparative study of zero-shot inference with large language models  and supervised modeling in breast cancer pathology classification

Internet of things (IoT) applications have become increasingly popular in
recent years, with applications ranging from building energy monitoring to
personal health tracking and activity recognition. In order to leverage these
data, automatic knowledge extraction - whereby we map from observations to
interpretable states and transitions - must be done at scale. As such, we have
seen many recent IoT data sets include annotations with a human expert
specifying states, recorded as a set of boundaries and associated labels in a
data sequence. These data can be used to build automatic labeling algorithms
that produce labels as an expert would. Here, we refer to human-specified
boundaries as breakpoints. Traditional changepoint detection methods only look
for statistically-detectable boundaries that are defined as abrupt variations
in the generative parameters of a data sequence. However, we observe that
breakpoints occur on more subtle boundaries that are non-trivial to detect with
these statistical methods. In this work, we propose a new unsupervised
approach, based on deep learning, that outperforms existing techniques and
learns the more subtle, breakpoint boundaries with a high accuracy. Through
extensive experiments on various real-world data sets - including
human-activity sensing data, speech signals, and electroencephalogram (EEG)
activity traces - we demonstrate the effectiveness of our algorithm for
practical applications. Furthermore, we show that our approach achieves
significantly better performance than previous methods.

本研究提出了一种新的基于深度学习的无监督方法，用于从 IoT 数据中映射出可解释的状态和转换。经过大量实验表明，该方法优于现有技术，并学习更微妙的断点边界。