The essence of multi-modal fusion lies in exploiting the complementary
information inherent in diverse modalities. However, prevalent fusion methods
rely on traditional neural architectures and are inadequately equipped to
capture the dynamics of interactions across modalities, particularly in
presence of complex intra- and inter-modality correlations. Recent advancements
in State Space Models (SSMs), notably exemplified by the Mamba model, have
emerged as promising contenders. Particularly, its state evolving process
implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an
appealing direction. However, fusing multiple modalities is challenging for
SSMs due to its hardware-aware parallelism designs. To this end, this paper
proposes the Coupled SSM model, for coupling state chains of multiple
modalities while maintaining independence of intra-modality state processes.
Specifically, in our coupled scheme, we devise an inter-modal hidden states
transition scheme, in which the current state is dependent on the states of its
own chain and that of the neighbouring chains at the previous time-step. To
fully comply with the hardware-aware parallelism, we devise an expedite coupled
state transition scheme and derive its corresponding global convolution kernel
for parallelism. Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through
multi-domain input verify the effectiveness of our model compared to current
state-of-the-art methods, improved F1-Score by 0.4\%, 0.9\%, and 2.3\% on the
three datasets respectively, 49\% faster inference and 83.7\% GPU memory save.
The results demonstrate that Coupled Mamba model is capable of enhanced
multi-modal fusion.

多模态融合的核心在于利用不同模态所固有的互补信息，然而现有的融合方法对于捕捉模态之间的复杂内部和相互关联动态缺乏充分准备，而最近在至关重要的状态空间模型 (SSM) 方面取得的进展，尤其是 Mamba 模型，成为有希望的竞争者，本文提出了耦合 SSM 模型来解决多种模态融合时的挑战，通过设计一个模态间隐藏状态转换方案以及并行计算的全局卷积核，实现多模态融合并在实验证实了其有效性。