Routine clinical visits of a patient produce not only image data, but also
non-image data containing clinical information regarding the patient, i.e.,
medical data is multi-modal in nature. Such heterogeneous modalities offer
different and complementary perspectives on the same patient, resulting in more
accurate clinical decisions when they are properly combined. However, despite
its significance, how to effectively fuse the multi-modal medical data into a
unified framework has received relatively little attention. In this paper, we
propose an effective graph-based framework called HetMed (Heterogeneous Graph
Learning for Multi-modal Medical Data Analysis) for fusing the multi-modal
medical data. Specifically, we construct a multiplex network that incorporates
multiple types of non-image features of patients to capture the complex
relationship between patients in a systematic way, which leads to more accurate
clinical decisions. Extensive experiments on various real-world datasets
demonstrate the superiority and practicality of HetMed. The source code for
HetMed is available at this https URL.

我们介绍一种名为 HetMed 的图形框架，可以将多模态医疗数据融合在一起，这通过构造一个多重网络来实现，该网络涵盖患者的多种非图像特征，以系统的方式捕捉患者之间的复杂关系，从而实现更准确的临床决策，并在各种实际数据集上证明了 HetMed 的优越性和实用性。

异构图学习在多模式医学数据分析中的应用

Heterogeneous Graph Learning for Multi-modal Medical Data Analysis

Question Answering on Electronic Health Records (EHR-QA) has a significant
impact on the healthcare domain, and it is being actively studied. Previous
research on structured EHR-QA focuses on converting natural language queries
into query language such as SQL or SPARQL (NLQ2Query), so the problem scope is
limited to pre-defined data types by the specific query language. In order to
expand the EHR-QA task beyond this limitation to handle multi-modal medical
data and solve complex inference in the future, more primitive systemic
language is needed. In this paper, we design the program-based model
(NLQ2Program) for EHR-QA as the first step towards the future direction. We
tackle MIMICSPARQL*, the graph-based EHR-QA dataset, via a program-based
approach in a semi-supervised manner in order to overcome the absence of gold
programs. Without the gold program, our proposed model shows comparable
performance to the previous state-of-the-art model, which is an NLQ2Query model
(0.9% gain). In addition, for a reliable EHR-QA model, we apply the uncertainty
decomposition method to measure the ambiguity in the input question. We
empirically confirmed data uncertainty is most indicative of the ambiguity in
the input question.

本文设计了程序为基础的模型（NLQ2Program）以利于未来拓展医疗多模态数据和解决复杂推理的电子健康记录问答任务，该模型在没有金标程序的情况下表现良好，运用不确定性分解方法可测度输入问题的模糊性。

针对结构化电子病历的问答不确定性感知文本转程序

Uncertainty-Aware Text-to-Program for Question Answering on Structured Electronic Health Records

Benefiting from the powerful expressive capability of graphs, graph-based
approaches have been popularly applied to handle multi-modal medical data and
achieved impressive performance in various biomedical applications. For disease
prediction tasks, most existing graph-based methods tend to define the graph
manually based on specified modality (e.g., demographic information), and then
integrated other modalities to obtain the patient representation by Graph
Representation Learning (GRL). However, constructing an appropriate graph in
advance is not a simple matter for these methods. Meanwhile, the complex
correlation between modalities is ignored. These factors inevitably yield the
inadequacy of providing sufficient information about the patient's condition
for a reliable diagnosis. To this end, we propose an end-to-end Multi-modal
Graph Learning framework (MMGL) for disease prediction with multi-modality. To
effectively exploit the rich information across multi-modality associated with
the disease, modality-aware representation learning is proposed to aggregate
the features of each modality by leveraging the correlation and complementarity
between the modalities. Furthermore, instead of defining the graph manually,
the latent graph structure is captured through an effective way of adaptive
graph learning. It could be jointly optimized with the prediction model, thus
revealing the intrinsic connections among samples. Our model is also applicable
to the scenario of inductive learning for those unseen data. An extensive group
of experiments on two disease prediction tasks demonstrates that the proposed
MMGL achieves more favorable performance. The code of MMGL is available at
https://github.com/SsGood/MMGL.

本研究提出了一种多模态图学习框架（MMGL）来预测疾病，该框架对每种模式的特征进行聚合，通过自适应图学习来捕获患者之间的内在联系，并在两个疾病预测任务上获得了更好的性能。

基于多模态图学习的疾病预测

Multi-modal Graph Learning for Disease Prediction

Recent advances in neuroscience have highlighted the effectiveness of
multi-modal medical data for investigating certain pathologies and
understanding human cognition. However, obtaining full sets of different
modalities is limited by various factors, such as long acquisition times, high
examination costs and artifact suppression. In addition, the complexity, high
dimensionality and heterogeneity of neuroimaging data remains another key
challenge in leveraging existing randomized scans effectively, as data of the
same modality is often measured differently by different machines. There is a
clear need to go beyond the traditional imaging-dependent process and
synthesize anatomically specific target-modality data from a source input. In
this paper, we propose to learn dedicated features that cross both intre- and
intra-modal variations using a novel CSC$\ell_4$Net. Through an initial
unification of intra-modal data in the feature maps and multivariate canonical
adaptation, CSC$\ell_4$Net facilitates feature-level mutual transformation. The
positive definite Riemannian manifold-penalized data fidelity term further
enables CSC$\ell_4$Net to reconstruct missing measurements according to
transformed features. Finally, the maximization $\ell_4$-norm boils down to a
computationally efficient optimization problem. Extensive experiments validate
the ability and robustness of our CSC$\ell_4$Net compared to the
state-of-the-art methods on multiple datasets.

本论文提出了一种名为 CSC l4 Net 的新型神经网络，它能在多模态医学数据中进行特征级相互转换和数据重构，表现优异。