Cross-modal transformers have demonstrated superiority in various vision
tasks by effectively integrating different modalities. This paper first
critiques prior token exchange methods which replace less informative tokens
with inter-modal features, and demonstrate exchange based methods underperform
cross-attention mechanisms, while the computational demand of the latter
inevitably restricts its use with longer sequences. To surmount the
computational challenges, we propose GeminiFusion, a pixel-wise fusion approach
that capitalizes on aligned cross-modal representations. GeminiFusion elegantly
combines intra-modal and inter-modal attentions, dynamically integrating
complementary information across modalities. We employ a layer-adaptive noise
to adaptively control their interplay on a per-layer basis, thereby achieving a
harmonized fusion process. Notably, GeminiFusion maintains linear complexity
with respect to the number of input tokens, ensuring this multimodal framework
operates with efficiency comparable to unimodal networks. Comprehensive
evaluations across multimodal image-to-image translation, 3D object detection
and arbitrary-modal semantic segmentation tasks, including RGB, depth, LiDAR,
event data, etc. demonstrate the superior performance of our GeminiFusion
against leading-edge techniques. The PyTorch code is available at
this https URL

跨模态变换器已经在不同的视觉任务中展示了优越性能，本文提出了一种像素级融合方法 GeminiFusion，它通过融合模态内部关注和模态间关注，动态地整合跨模态的互补信息，同时通过层自适应噪声实现了协调的融合过程，该方法在多模态图像转换、三维物体检测和任意模态的语义分割任务中表现出优秀的性能。