Recent advancements in non-invasive detection of cardiac hemodynamic
instability (CHDI) primarily focus on applying machine learning techniques to a
single data modality, e.g. cardiac magnetic resonance imaging (MRI). Despite
their potential, these approaches often fall short especially when the size of
labeled patient data is limited, a common challenge in the medical domain.
Furthermore, only a few studies have explored multimodal methods to study CHDI,
which mostly rely on costly modalities such as cardiac MRI and echocardiogram.
In response to these limitations, we propose a novel multimodal variational
autoencoder ($\text{CardioVAE}_\text{X,G}$) to integrate low-cost chest X-ray
(CXR) and electrocardiogram (ECG) modalities with pre-training on a large
unlabeled dataset. Specifically, $\text{CardioVAE}_\text{X,G}$ introduces a
novel tri-stream pre-training strategy to learn both shared and
modality-specific features, thus enabling fine-tuning with both unimodal and
multimodal datasets. We pre-train $\text{CardioVAE}_\text{X,G}$ on a large,
unlabeled dataset of $50,982$ subjects from a subset of MIMIC database and then
fine-tune the pre-trained model on a labeled dataset of $795$ subjects from the
ASPIRE registry. Comprehensive evaluations against existing methods show that
$\text{CardioVAE}_\text{X,G}$ offers promising performance (AUROC $=0.79$ and
Accuracy $=0.77$), representing a significant step forward in non-invasive
prediction of CHDI. Our model also excels in producing fine interpretations of
predictions directly associated with clinical features, thereby supporting
clinical decision-making.

利用低成本的胸部 X 光和心电图模态，结合大规模未标记数据的预训练方法，我们提出了一种新颖的多模态变分自编码器（CardioVAE_X,G），以实现对心脏血流动力学不稳定性（CHDI）的无创预测，在预测 CHDI 方面显示出有希望的性能，对相关的临床特征解释上也表现出色，从而支持临床决策。