Human Activity Recognition is a longstanding problem in AI with applications
in a broad range of areas: from healthcare, sports and fitness, security, and
human computer interaction to robotics. The performance of HAR in real-world
settings is strongly dependent on the type and quality of the input signal that
can be acquired. Given an unobstructed, high-quality camera view of a scene,
computer vision systems, in particular in conjunction with foundational models
(e.g., CLIP), can today fairly reliably distinguish complex activities. On the
other hand, recognition using modalities such as wearable sensors (which are
often more broadly available, e.g, in mobile phones and smartwatches) is a more
difficult problem, as the signals often contain less information and labeled
training data is more difficult to acquire. In this work, we show how we can
improve HAR performance across different modalities using multimodal
contrastive pretraining. Our approach MuJo (Multimodal Joint Feature Space
Learning), learns a multimodal joint feature space with video, language, pose,
and IMU sensor data. The proposed approach combines contrastive and multitask
learning methods and analyzes different multitasking strategies for learning a
compact shared representation. A large dataset with parallel video, language,
pose, and sensor data points is also introduced to support the research, along
with an analysis of the robustness of the multimodal joint space for
modal-incomplete and low-resource data. On the MM-Fit dataset, our model
achieves an impressive Macro F1-Score of up to 0.992 with only 2% of the train
data and 0.999 when using all available training data for classification tasks.
Moreover, in the scenario where the MM-Fit dataset is unseen, we demonstrate a
generalization performance of up to 0.638.

通过多模态对比预训练方法 MuJo，利用视频、语言、姿势和 IMU 传感器数据，改善了不同模态下的人类活动识别性能，在 MM-Fit 数据集上达到了令人印象深刻的宏平均 F1-Score 为 0.992 和 0.999 的分类效果，同时展示了最高达 0.638 的泛化性能。

MuJo：人体活动识别的多模态联合特征空间学习

MuJo: Multimodal Joint Feature Space Learning for Human Activity  Recognition

Multimodal contrastive pretraining has been used to train multimodal
representation models, such as CLIP, on large amounts of paired image-text
data. However, previous studies have revealed that such models are vulnerable
to backdoor attacks. Specifically, when trained on backdoored examples, CLIP
learns spurious correlations between the embedded backdoor trigger and the
target label, aligning their representations in the joint embedding space.
Injecting even a small number of poisoned examples, such as 75 examples in 3
million pretraining data, can significantly manipulate the model's behavior,
making it difficult to detect or unlearn such correlations. To address this
issue, we propose CleanCLIP, a finetuning framework that weakens the learned
spurious associations introduced by backdoor attacks by independently
re-aligning the representations for individual modalities. We demonstrate that
unsupervised finetuning using a combination of multimodal contrastive and
unimodal self-supervised objectives for individual modalities can significantly
reduce the impact of the backdoor attack. Additionally, we show that supervised
finetuning on task-specific labeled image data removes the backdoor trigger
from the CLIP vision encoder. We show empirically that CleanCLIP maintains
model performance on benign examples while erasing a range of backdoor attacks
on multimodal contrastive learning.

介绍了一种解决多模态表征学习领域中后门攻击问题的新方法 CleanCLIP，可以帮助模型减弱后门数据带来的错误训练结果。同时，该方法可以通过对单个模态的预训练任务进行微调的方式来提高模型的鲁棒性。