Within the intensive care unit (ICU), a wealth of patient data, including
clinical measurements and clinical notes, is readily available. This data is a
valuable resource for comprehending patient health and informing medical
decisions, but it also contains many challenges in analysis. Deep learning
models show promise in extracting meaningful patterns, but they require
extensive labeled data, a challenge in critical care. To address this, we
propose a novel approach employing self-supervised pretraining, focusing on the
alignment of clinical measurements and notes. Our approach combines contrastive
and masked token prediction tasks during pretraining. Semi-supervised
experiments on the MIMIC-III dataset demonstrate the effectiveness of our
self-supervised pretraining. In downstream tasks, including in-hospital
mortality prediction and phenotyping, our pretrained model outperforms
baselines in settings where only a fraction of the data is labeled, emphasizing
its ability to enhance ICU data analysis. Notably, our method excels in
situations where very few labels are available, as evidenced by an increase in
the AUC-ROC for in-hospital mortality by 0.17 and in AUC-PR for phenotyping by
0.1 when only 1% of labels are accessible. This work advances self-supervised
learning in the healthcare domain, optimizing clinical insights from abundant
yet challenging ICU data.

在重症监护室内，包括临床测量和临床笔记在内的丰富患者数据可供利用，但其分析也面临许多挑战。本研究提出了一种新颖的方法，利用自我监督预训练来解决深度学习模型在重症监护中的标注数据不足问题，并在数据有限的情况下，在院内死亡预测和表型预测等下游任务中优于基线模型，突显其提升重症监护室数据分析能力。

医学时间序列和笔记的多模态预训练

Multimodal Pretraining of Medical Time Series and Notes

Intensive Care in-hospital mortality prediction has various clinical
applications. Neural prediction models, especially when capitalising on
clinical notes, have been put forward as improvement on currently existing
models. However, to be acceptable these models should be performant and
transparent. This work studies different attention mechanisms for clinical
neural prediction models in terms of their discrimination and calibration.
Specifically, we investigate sparse attention as an alternative to dense
attention weights in the task of in-hospital mortality prediction from clinical
notes. We evaluate the attention mechanisms based on: i) local self-attention
over words in a sentence, and ii) global self-attention with a transformer
architecture across sentences. We demonstrate that the sparse mechanism
approach outperforms the dense one for the local self-attention in terms of
predictive performance with a publicly available dataset, and puts higher
attention to prespecified relevant directive words. The performance at the
sentence level, however, deteriorates as sentences including the influential
directive words tend to be dropped all together.

本文研究了临床神经预测模型中的不同注意力机制在区分度和校准性方面的表现，特别是将稀疏注意力作为在临床记录中预测住院病死率的替代方法，在公开数据集上表现出了优异的预测性能，并且在确定的相关指令词上更加注意力集中。然而，在句子选择上可能会出现问题。

使用字级别和句级别稀疏注意力的临床笔记死亡率预测模型

Mortality Prediction Models with Clinical Notes Using Sparse Attention at the Word and Sentence Levels

Machine learning models depend on the quality of input data. As electronic
health records are widely adopted, the amount of data in health care is
growing, along with complaints about the quality of medical notes. We use two
prediction tasks, readmission prediction and in-hospital mortality prediction,
to characterize the value of information in medical notes. We show that as a
whole, medical notes only provide additional predictive power over structured
information in readmission prediction. We further propose a probing framework
to select parts of notes that enable more accurate predictions than using all
notes, despite that the selected information leads to a distribution shift from
the training data ("all notes"). Finally, we demonstrate that models trained on
the selected valuable information achieve even better predictive performance,
with only 6.8% of all the tokens for readmission prediction.

使用两项预测任务，即再入院预测和入院期间死亡预测，来表征医疗笔记中信息的价值，我们发现整体而言，医疗笔记仅在再入院预测中提供额外的预测能力。最后，我们证明，针对所选有价值的信息进行训练的模型可以实现更好的预测性能，而只使用 6.8％的所有标记即可实现。