In this paper, we introduce a novel Synchronized Class Token Fusion (SCT
Fusion) architecture in the framework of multi-modal multi-label classification
(MLC) of remote sensing (RS) images. The proposed architecture leverages
modality-specific attention-based transformer encoders to process varying input
modalities, while exchanging information across modalities by synchronizing the
special class tokens after each transformer encoder block. The synchronization
involves fusing the class tokens with a trainable fusion transformation,
resulting in a synchronized class token that contains information from all
modalities. As the fusion transformation is trainable, it allows to reach an
accurate representation of the shared features among different modalities.
Experimental results show the effectiveness of the proposed architecture over
single-modality architectures and an early fusion multi-modal architecture when
evaluated on a multi-modal MLC dataset.
The code of the proposed architecture is publicly available at
this https URL

介绍了一种新的带有同步类令牌融合（SCT Fusion）的架构，用于处理多模态遥感（RS）图像的多模态多标签分类（MLC），该架构利用了模态特定的注意力机制，通过在每个变压器编码器块之后同步类令牌实现跨模态交换信息。