The mechanism of connecting multimodal signals through self-attention
operation is a key factor in the success of multimodal Transformer networks in
remote sensing data fusion tasks. However, traditional approaches assume access
to all modalities during both training and inference, which can lead to severe
degradation when dealing with modal-incomplete inputs in downstream
applications. To address this limitation, our proposed approach introduces a
novel model for incomplete multimodal learning in the context of remote sensing
data fusion. This approach can be used in both supervised and self-supervised
pretraining paradigms and leverages the additional learned fusion tokens in
combination with Bi-LSTM attention and masked self-attention mechanisms to
collect multimodal signals. The proposed approach employs reconstruction and
contrastive loss to facilitate fusion in pre-training while allowing for random
modality combinations as inputs in network training. Our approach delivers
state-of-the-art performance on two multimodal datasets for tasks such as
building instance / semantic segmentation and land-cover mapping tasks when
dealing with incomplete inputs during inference.

本文提出了一种新颖的不完整多模态学习模型，结合双向 LSTM 注意力和掩蔽自注意机制来收集多模态信号，利用重构和对比损失来促进预训练中的融合，能在处理数据不完整输入时达到当前最佳性能水平。

遥感数据融合的不完全多模态学习

Incomplete Multimodal Learning for Remote Sensing Data Fusion

Decoding emotional states from human brain activity plays an important role
in brain-computer interfaces. Existing emotion decoding methods still have two
main limitations: one is only decoding a single emotion category from a brain
activity pattern and the decoded emotion categories are coarse-grained, which
is inconsistent with the complex emotional expression of human; the other is
ignoring the discrepancy of emotion expression between the left and right
hemispheres of human brain. In this paper, we propose a novel multi-view
multi-label hybrid model for fine-grained emotion decoding (up to 80 emotion
categories) which can learn the expressive neural representations and
predicting multiple emotional states simultaneously. Specifically, the
generative component of our hybrid model is parametrized by a multi-view
variational auto-encoder, in which we regard the brain activity of left and
right hemispheres and their difference as three distinct views, and use the
product of expert mechanism in its inference network. The discriminative
component of our hybrid model is implemented by a multi-label classification
network with an asymmetric focal loss. For more accurate emotion decoding, we
first adopt a label-aware module for emotion-specific neural representations
learning and then model the dependency of emotional states by a masked
self-attention mechanism. Extensive experiments on two visually evoked
emotional datasets show the superiority of our method.

本文提出了一种用于通过大脑活动数据进行情绪识别的新型多视图、多标签的混合模型，可以同时学习表达性的神经表示并多元预测情绪状态，通过实验表明该方法具有较高的识别准确率。