Human activity recognition (HAR) has been playing an increasingly important
role in various domains such as healthcare, security monitoring, and metaverse
gaming. Though numerous HAR methods based on computer vision have been
developed to show prominent performance, they still suffer from poor robustness
in adverse visual conditions in particular low illumination, which motivates
WiFi-based HAR to serve as a good complementary modality. Existing solutions
using WiFi and vision modalities rely on massive labeled data that are very
cumbersome to collect. In this paper, we propose a novel unsupervised
multimodal HAR solution, MaskFi, that leverages only unlabeled video and WiFi
activity data for model training. We propose a new algorithm, masked
WiFi-vision modeling (MI2M), that enables the model to learn cross-modal and
single-modal features by predicting the masked sections in representation
learning. Benefiting from our unsupervised learning procedure, the network
requires only a small amount of annotated data for finetuning and can adapt to
the new environment with better performance. We conduct extensive experiments
on two WiFi-vision datasets collected in-house, and our method achieves human
activity recognition and human identification in terms of both robustness and
accuracy.

我们提出了一种新的无监督多模态 HAR 解决方案 MaskFi，该方案利用仅有的未标记视频和 WiFi 活动数据进行模型训练，通过预测表示学习中的遮挡区域，使模型能够学习跨模态和单模态特征。我们的方法在两个内部采集的 WiFi-vision 数据集上进行了广泛的实验证明，同时在鲁棒性和准确性方面实现了人类活动识别和人员识别。