Due to the distinctive characteristics of sensors, each modality exhibits
unique physical properties. For this reason, in the context of multi-modal
action recognition, it is important to consider not only the overall action
content but also the complementary nature of different modalities. In this
paper, we propose a novel network, named Modality Mixer (M-Mixer) network,
which effectively leverages and incorporates the complementary information
across modalities with the temporal context of actions for action recognition.
A key component of our proposed M-Mixer is the Multi-modal Contextualization
Unit (MCU), a simple yet effective recurrent unit. Our MCU is responsible for
temporally encoding a sequence of one modality (e.g., RGB) with action content
features of other modalities (e.g., depth and infrared modalities). This
process encourages M-Mixer network to exploit global action content and also to
supplement complementary information of other modalities. Furthermore, to
extract appropriate complementary information regarding to the given modality
settings, we introduce a new module, named Complementary Feature Extraction
Module (CFEM). CFEM incorporates sepearte learnable query embeddings for each
modality, which guide CFEM to extract complementary information and global
action content from the other modalities. As a result, our proposed method
outperforms state-of-the-art methods on NTU RGB+D 60, NTU RGB+D 120, and
NW-UCLA datasets. Moreover, through comprehensive ablation studies, we further
validate the effectiveness of our proposed method.

提出了一种名为 Modality Mixer (M-Mixer) 网络的新型网络，有效地利用和整合不同模态的互补信息和动作的时间上下文，以实现动作识别，通过在 NTU RGB+D 60、NTU RGB+D 120 和 NW-UCLA 数据集上的测试结果，证明了该方法优于最先进的方法。