In this work we introduce Labrador, a pre-trained Transformer model for
laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million
lab test results from electronic health records (EHRs) and evaluated on various
downstream outcome prediction tasks. Both models demonstrate mastery of the
pre-training task but neither consistently outperform XGBoost on downstream
supervised tasks. Our ablation studies reveal that transfer learning shows
limited effectiveness for BERT and achieves marginal success with Labrador. We
explore the reasons for the failure of transfer learning and suggest that the
data generating process underlying each patient cannot be characterized
sufficiently using labs alone, among other factors. We encourage future work to
focus on joint modeling of multiple EHR data categories and to include
tree-based baselines in their evaluations.

我们介绍了一种名为 Labrador 的预训练 Transformers 模型，用于实验室数据的分析，并在预测各种下游结果任务中对 Labrador 和 BERT 进行了评估。两种模型在预训练任务上表现出了熟练的水平，但在下游的有监督任务中，两者都无法持续胜过 XGBoost。我们的剖析研究揭示了对于 BERT 而言，迁移学习的效果有限，并且 Labrador 仅取得了较小的成功。我们探索了迁移学习失败的原因，并认为不能仅通过实验室数据来充分表征每个患者的数据生成过程，其中还有其他因素。我们鼓励未来的研究将重点放在多个电子健康记录数据类别的联合建模上，并在评估中包括基于树的基准模型。