Driver action recognition has significantly advanced in enhancing
driver-vehicle interactions and ensuring driving safety by integrating multiple
modalities, such as infrared and depth. Nevertheless, compared to RGB modality
only, it is always laborious and costly to collect extensive data for all types
of non-RGB modalities in car cabin environments. Therefore, previous works have
suggested independently learning each non-RGB modality by fine-tuning a model
pre-trained on RGB videos, but these methods are less effective in extracting
informative features when faced with newly-incoming modalities due to large
domain gaps. In contrast, we propose a Continual Cross-Modal Mapping Network
(CM2-Net) to continually learn each newly-incoming modality with instructive
prompts from the previously-learned modalities. Specifically, we have developed
Accumulative Cross-modal Mapping Prompting (ACMP), to map the discriminative
and informative features learned from previous modalities into the feature
space of newly-incoming modalities. Then, when faced with newly-incoming
modalities, these mapped features are able to provide effective prompts for
which features should be extracted and prioritized. These prompts are
accumulating throughout the continual learning process, thereby boosting
further recognition performances. Extensive experiments conducted on the
Drive&Act dataset demonstrate the performance superiority of CM2-Net on both
uni- and multi-modal driver action recognition.

在司机行为识别方面，我们提出了 Continual Cross-Modal Mapping Network (CM2-Net) 来不断学习各种新的模态，并利用 Accumulative Cross-modal Mapping Prompting (ACMP) 来提升识别性能。在 Drive&Act 数据集上的实验证明了 CM2-Net 在单模态和多模态司机行为识别上的卓越性能。

CM2-Net: 司机动作识别的持续跨模态映射网络

CM2-Net: Continual Cross-Modal Mapping Network for Driver Action  Recognition

Driver Action Recognition (DAR) is crucial in vehicle cabin monitoring
systems. In real-world applications, it is common for vehicle cabins to be
equipped with cameras featuring different modalities. However, multi-modality
fusion strategies for the DAR task within car cabins have rarely been studied.
In this paper, we propose a novel yet efficient multi-modality driver action
recognition method based on dual feature shift, named DFS. DFS first integrates
complementary features across modalities by performing modality feature
interaction. Meanwhile, DFS achieves the neighbour feature propagation within
single modalities, by feature shifting among temporal frames. To learn common
patterns and improve model efficiency, DFS shares feature extracting stages
among multiple modalities. Extensive experiments have been carried out to
verify the effectiveness of the proposed DFS model on the Drive\&Act dataset.
The results demonstrate that DFS achieves good performance and improves the
efficiency of multi-modality driver action recognition.

基于双特征位移的新型高效多模态驾驶员动作识别方法 DFS，通过模态特征相互作用和时间帧之间的特征位移实现跨模态综合特征，共享多个模态中的特征提取阶段，以学习常见模式并提高模型效率，在 Drive&Act 数据集上验证了 DFS 模型的有效性和优良性能。