In this work we introduce Labrador, a pre-trained Transformer model for
laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million
lab test results from electronic health records (EHRs) and evaluated on various
downstream outcome prediction tasks. Both models demonstrate mastery of the
pre-training task but neither consistently outperform XGBoost on downstream
supervised tasks. Our ablation studies reveal that transfer learning shows
limited effectiveness for BERT and achieves marginal success with Labrador. We
explore the reasons for the failure of transfer learning and suggest that the
data generating process underlying each patient cannot be characterized
sufficiently using labs alone, among other factors. We encourage future work to
focus on joint modeling of multiple EHR data categories and to include
tree-based baselines in their evaluations.

我们介绍了一种名为 Labrador 的预训练 Transformers 模型，用于实验室数据的分析，并在预测各种下游结果任务中对 Labrador 和 BERT 进行了评估。两种模型在预训练任务上表现出了熟练的水平，但在下游的有监督任务中，两者都无法持续胜过 XGBoost。我们的剖析研究揭示了对于 BERT 而言，迁移学习的效果有限，并且 Labrador 仅取得了较小的成功。我们探索了迁移学习失败的原因，并认为不能仅通过实验室数据来充分表征每个患者的数据生成过程，其中还有其他因素。我们鼓励未来的研究将重点放在多个电子健康记录数据类别的联合建模上，并在评估中包括基于树的基准模型。

Labrador: 探索实验室数据的被屏蔽语言模型的极限

Labrador: Exploring the Limits of Masked Language Modeling for  Laboratory Data

We study the phenomenon of \textit{in-context learning} (ICL) exhibited by
large language models, where they can adapt to a new learning task, given a
handful of labeled examples, without any explicit parameter optimization. Our
goal is to explain how a pre-trained transformer model is able to perform ICL
under reasonable assumptions on the pre-training process and the downstream
tasks. We posit a mechanism whereby a transformer can achieve the following:
(a) receive an i.i.d. sequence of examples which have been converted into a
prompt using potentially-ambiguous delimiters, (b) correctly segment the prompt
into examples and labels, (c) infer from the data a \textit{sparse linear
regressor} hypothesis, and finally (d) apply this hypothesis on the given test
example and return a predicted label. We establish that this entire procedure
is implementable using the transformer mechanism, and we give sample complexity
guarantees for this learning framework. Our empirical findings validate the
challenge of segmentation, and we show a correspondence between our posited
mechanisms and observed attention maps for step (c).

本文研究了大型语言模型展示的 “上下文学习” 现象，并解释了预训练的 transformer 模型如何在合理的假设下执行上下文学习。我们推出了一种机制，使得 transformer 模型能够正确划分上下文，推断出稀疏线性回归假说，并应用此假说进行预测，在该学习框架中的样本复杂度保证。

用于稀疏检索任务的样本高效上下文学习机制

A Mechanism for Sample-Efficient In-Context Learning for Sparse  Retrieval Tasks

Infectious diseases are a significant public health concern globally, and
extracting relevant information from scientific literature can facilitate the
development of effective prevention and treatment strategies. However, the
large amount of clinical data available presents a challenge for information
extraction. To address this challenge, this study proposes a natural language
processing (NLP) framework that uses a pre-trained transformer model fine-tuned
on task-specific data to extract key information related to infectious diseases
from free-text clinical data. The proposed framework includes three components:
a data layer for preparing datasets from clinical texts, a foundation model
layer for entity extraction, and an assessment layer for performance analysis.
The results of the evaluation indicate that the proposed method outperforms
standard methods, and leveraging prior knowledge through the pre-trained
transformer model makes it useful for investigating other infectious diseases
in the future.

本研究提出了一种自然语言处理 (NLP) 框架，使用在任务特定数据上微调的预训练变压器模型来从自由文本临床数据中提取与传染性疾病相关的关键信息，并且结果表明该方法性能优异。