Human activity recognition (HAR) will be an essential function of various
emerging applications. However, HAR typically encounters challenges related to
modality limitations and label scarcity, leading to an application gap between
current solutions and real-world requirements. In this work, we propose MESEN,
a multimodal-empowered unimodal sensing framework, to utilize unlabeled
multimodal data available during the HAR model design phase for unimodal HAR
enhancement during the deployment phase. From a study on the impact of
supervised multimodal fusion on unimodal feature extraction, MESEN is designed
to feature a multi-task mechanism during the multimodal-aided pre-training
stage. With the proposed mechanism integrating cross-modal feature contrastive
learning and multimodal pseudo-classification aligning, MESEN exploits
unlabeled multimodal data to extract effective unimodal features for each
modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a
few labeled samples. Extensive experiments on eight public multimodal datasets
demonstrate that MESEN achieves significant performance improvements over
state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal
data.

通过利用多模态数据来增强单模态人体活动识别的效果，MESEN 实现了显著的性能提升，解决了当前解决方案与实际应用需求之间的差距。