Improving the performance of semantic segmentation models using multispectral
information is crucial, especially for environments with low-light and adverse
conditions. Multi-modal fusion techniques pursue either the learning of
cross-modality features to generate a fused image or engage in knowledge
distillation but address multimodal and missing modality scenarios as distinct
issues, which is not an optimal approach for multi-sensor models. To address
this, a novel multi-modal fusion approach called CSK-Net is proposed, which
uses a contrastive learning-based spectral knowledge distillation technique
along with an automatic mixed feature exchange mechanism for semantic
segmentation in optical (EO) and infrared (IR) images. The distillation scheme
extracts detailed textures from the optical images and distills them into the
optical branch of CSK-Net. The model encoder consists of shared convolution
weights with separate batch norm (BN) layers for both modalities, to capture
the multi-spectral information from different modalities of the same objects. A
Novel Gated Spectral Unit (GSU) and mixed feature exchange strategy are
proposed to increase the correlation of modality-shared information and
decrease the modality-specific information during the distillation process.
Comprehensive experiments show that CSK-Net surpasses state-of-the-art models
in multi-modal tasks and for missing modalities when exclusively utilizing IR
data for inference across three public benchmarking datasets. For missing
modality scenarios, the performance increase is achieved without additional
computational costs compared to the baseline segmentation models.

通过使用多光谱信息来提高语义分割模型的性能对于低光和恶劣环境至关重要。提出了一种名为 CSK-Net 的新型多模态融合方法，它利用对比学习为光学（EO）和红外（IR）图像的语义分割提供了基于光谱知识蒸馏的融合技术。该方法不仅在多模态任务上超过了现有的模型，而且在缺失模态的情况下，仅利用 IR 数据进行推断就能提高性能，而与基线分割模型相比，并没有额外的计算成本。