Scene understanding based on image segmentation is a crucial component for
autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be
advanced by exploiting informative features from the supplementary modality
(X-modality). In this work, we propose CMX, a transformer-based cross-modal
fusion framework for RGB-X semantic segmentation. To generalize to different
sensing modalities encompassing various supplements and uncertainties, we
consider that comprehensive cross-modal interactions should be provided. CMX is
built with two streams to extract features from RGB images and the X-modality.
In each feature extraction stage, we design a Cross-Modal Feature Rectification
Module (CM-FRM) to calibrate the feature of the current modality by combining
the feature from the other modality, in spatial- and channel-wise dimensions.
With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix
them for the final semantic prediction. FFM is constructed with a
cross-attention mechanism, which enables exchange of long-range contexts,
enhancing bi-modal features globally. Extensive experiments show that CMX
generalizes to diverse multi-modal combinations, achieving state-of-the-art
performances on five RGB-Depth benchmarks, as well as RGB-Thermal,
RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the
generalizability to dense-sparse data fusion, we establish an RGB-Event
semantic segmentation benchmark based on the EventScape dataset, on which CMX
sets the new state-of-the-art. The source code of CMX is publicly available at
this https URL

本文提出了一种基于转换器的跨模态融合框架 CMX，用于针对自主车辆的 RGB-X 语义分割，结果表明 CMX 可以推广到不同的感知模态，并实现在多个基准数据集上的最新技术性能。