Human Activity Recognition is a longstanding problem in AI with applications
in a broad range of areas: from healthcare, sports and fitness, security, and
human computer interaction to robotics. The performance of HAR in real-world
settings is strongly dependent on the type and quality of the input signal that
can be acquired. Given an unobstructed, high-quality camera view of a scene,
computer vision systems, in particular in conjunction with foundational models
(e.g., CLIP), can today fairly reliably distinguish complex activities. On the
other hand, recognition using modalities such as wearable sensors (which are
often more broadly available, e.g, in mobile phones and smartwatches) is a more
difficult problem, as the signals often contain less information and labeled
training data is more difficult to acquire. In this work, we show how we can
improve HAR performance across different modalities using multimodal
contrastive pretraining. Our approach MuJo (Multimodal Joint Feature Space
Learning), learns a multimodal joint feature space with video, language, pose,
and IMU sensor data. The proposed approach combines contrastive and multitask
learning methods and analyzes different multitasking strategies for learning a
compact shared representation. A large dataset with parallel video, language,
pose, and sensor data points is also introduced to support the research, along
with an analysis of the robustness of the multimodal joint space for
modal-incomplete and low-resource data. On the MM-Fit dataset, our model
achieves an impressive Macro F1-Score of up to 0.992 with only 2% of the train
data and 0.999 when using all available training data for classification tasks.
Moreover, in the scenario where the MM-Fit dataset is unseen, we demonstrate a
generalization performance of up to 0.638.

通过多模态对比预训练方法 MuJo，利用视频、语言、姿势和 IMU 传感器数据，改善了不同模态下的人类活动识别性能，在 MM-Fit 数据集上达到了令人印象深刻的宏平均 F1-Score 为 0.992 和 0.999 的分类效果，同时展示了最高达 0.638 的泛化性能。