The mechanism of connecting multimodal signals through self-attention
operation is a key factor in the success of multimodal Transformer networks in
remote sensing data fusion tasks. However, traditional approaches assume access
to all modalities during both training and inference, which can lead to severe
degradation when dealing with modal-incomplete inputs in downstream
applications. To address this limitation, our proposed approach introduces a
novel model for incomplete multimodal learning in the context of remote sensing
data fusion. This approach can be used in both supervised and self-supervised
pretraining paradigms and leverages the additional learned fusion tokens in
combination with Bi-LSTM attention and masked self-attention mechanisms to
collect multimodal signals. The proposed approach employs reconstruction and
contrastive loss to facilitate fusion in pre-training while allowing for random
modality combinations as inputs in network training. Our approach delivers
state-of-the-art performance on two multimodal datasets for tasks such as
building instance / semantic segmentation and land-cover mapping tasks when
dealing with incomplete inputs during inference.

本文提出了一种新颖的不完整多模态学习模型，结合双向 LSTM 注意力和掩蔽自注意机制来收集多模态信号，利用重构和对比损失来促进预训练中的融合，能在处理数据不完整输入时达到当前最佳性能水平。

遥感数据融合的不完全多模态学习

Incomplete Multimodal Learning for Remote Sensing Data Fusion

Conversations have become a critical data format on social media platforms.
Understanding conversation from emotion, content and other aspects also
attracts increasing attention from researchers due to its widespread
application in human-computer interaction. In real-world environments, we often
encounter the problem of incomplete modalities, which has become a core issue
of conversation understanding. To address this problem, researchers propose
various methods. However, existing approaches are mainly designed for
individual utterances rather than conversational data, which cannot fully
exploit temporal and speaker information in conversations. To this end, we
propose a novel framework for incomplete multimodal learning in conversations,
called "Graph Complete Network (GCNet)", filling the gap of existing works. Our
GCNet contains two well-designed graph neural network-based modules, "Speaker
GNN" and "Temporal GNN", to capture temporal and speaker dependencies. To make
full use of complete and incomplete data, we jointly optimize classification
and reconstruction tasks in an end-to-end manner. To verify the effectiveness
of our method, we conduct experiments on three benchmark conversational
datasets. Experimental results demonstrate that our GCNet is superior to
existing state-of-the-art approaches in incomplete multimodal learning. Code is
available at this https URL.

该研究提出了一种用于在对话中进行不完整多模态学习的新型框架，名为 “Graph Complete Network (GCNet)”，包含两个基于图神经网络的模块 'Speaker GNN' 和 'Temporal GNN' 来捕捉时间和说话者间的依赖关系。实验结果表明，相比于现有的最先进方法，我们的 GCNet 在不完整多模态学习方面表现得更好。