Multimodal electronic health record (EHR) data can offer a holistic
assessment of a patient's health status, supporting various predictive
healthcare tasks. Recently, several studies have embraced the multitask
learning approach in the healthcare domain, exploiting the inherent
correlations among clinical tasks to predict multiple outcomes simultaneously.
However, existing methods necessitate samples to possess complete labels for
all tasks, which places heavy demands on the data and restricts the flexibility
of the model. Meanwhile, within a multitask framework with multimodal inputs,
how to comprehensively consider the information disparity among modalities and
among tasks still remains a challenging problem. To tackle these issues, a
unified healthcare prediction model, also named by \textbf{FlexCare}, is
proposed to flexibly accommodate incomplete multimodal inputs, promoting the
adaption to multiple healthcare tasks. The proposed model breaks the
conventional paradigm of parallel multitask prediction by decomposing it into a
series of asynchronous single-task prediction. Specifically, a task-agnostic
multimodal information extraction module is presented to capture decorrelated
representations of diverse intra- and inter-modality patterns. Taking full
account of the information disparities between different modalities and
different tasks, we present a task-guided hierarchical multimodal fusion module
that integrates the refined modality-level representations into an individual
patient-level representation. Experimental results on multiple tasks from
MIMIC-IV/MIMIC-CXR/MIMIC-NOTE datasets demonstrate the effectiveness of the
proposed method. Additionally, further analysis underscores the feasibility and
potential of employing such a multitask strategy in the healthcare domain. The
source code is available at this https URL

提出了一种名为 FlexCare 的统一医疗预测模型，可以灵活地处理不完整的多模态输入，改善对多个医疗任务的适应性。该模型通过将多任务预测分解为一系列异步的单任务预测来突破传统的并行多任务预测范式，并通过引入多模态信息提取模块和多模态融合模块，在考虑不同模态之间和不同任务之间的信息差异的基础上实现对全面多模态数据的利用。通过在多个任务上进行实验，并进一步分析，验证了该多任务策略在医疗领域的可行性和潜力。

FlexCare: 灵活的多模态健康预测中利用交叉任务协同

FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal  Healthcare Prediction

RGB thermal scene parsing has recently attracted increasing research interest
in the field of computer vision. However, most existing methods fail to perform
good boundary extraction for prediction maps and cannot fully use high level
features. In addition, these methods simply fuse the features from RGB and
thermal modalities but are unable to obtain comprehensive fused features. To
address these problems, we propose an edge-aware guidance fusion network
(EGFNet) for RGB thermal scene parsing. First, we introduce a prior edge map
generated using the RGB and thermal images to capture detailed information in
the prediction map and then embed the prior edge information in the feature
maps. To effectively fuse the RGB and thermal information, we propose a
multimodal fusion module that guarantees adequate cross-modal fusion.
Considering the importance of high level semantic information, we propose a
global information module and a semantic information module to extract rich
semantic information from the high-level features. For decoding, we use simple
elementwise addition for cascaded feature fusion. Finally, to improve the
parsing accuracy, we apply multitask deep supervision to the semantic and
boundary maps. Extensive experiments were performed on benchmark datasets to
demonstrate the effectiveness of the proposed EGFNet and its superior
performance compared with state of the art methods. The code and results can be
found at this https URL

提出一种边缘感知融合引导网络（EGFNet）用于 RGB 热场景解析，在 RGB 和热图像上生成先验边缘映射捕捉详细信息，使用多模态融合模块来融合 RGB 和热信息，使用全局信息和语义信息模块提取高层次特征的语义信息，应用多任务深度监督来提高解析准确性，实验结果表明 EGFNet 是优于现有方法的。

RGB 热图场景分割的边缘感知引导融合网络

Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing

We consider the problem of Visual Question Answering (VQA). Given an image
and a free-form, open-ended, question, expressed in natural language, the goal
of VQA system is to provide accurate answer to this question with respect to
the image. The task is challenging because it requires simultaneous and
intricate understanding of both visual and textual information. Attention,
which captures intra- and inter-modal dependencies, has emerged as perhaps the
most widely used mechanism for addressing these challenges. In this paper, we
propose an improved attention-based architecture to solve VQA. We incorporate
an Attention on Attention (AoA) module within encoder-decoder framework, which
is able to determine the relation between attention results and queries.
Attention module generates weighted average for each query. On the other hand,
AoA module first generates an information vector and an attention gate using
attention results and current context; and then adds another attention to
generate final attended information by multiplying the two. We also propose
multimodal fusion module to combine both visual and textual information. The
goal of this fusion module is to dynamically decide how much information should
be considered from each modality. Extensive experiments on VQA-v2 benchmark
dataset show that our method achieves the state-of-the-art performance.

本文提出了一种改进的基于注意力机制的架构，其中包括一个对注意力结果和查询进行关系确定的 AoA 模块，并提出了多模态融合模块来组合视觉和文本信息，结果在 VQA-v2 基准数据集上达到了最先进的性能。