Due to the scarcity of labeled sensor data in HAR, prior research has turned
to video data to synthesize Inertial Measurement Units (IMU) data, capitalizing
on its rich activity annotations. However, generating IMU data from videos
presents challenges for HAR in real-world settings, attributed to the poor
quality of synthetic IMU data and its limited efficacy in subtle, fine-grained
motions. In this paper, we propose Multi$^3$Net, our novel multi-modal,
multitask, and contrastive-based framework approach to address the issue of
limited data. Our pretraining procedure uses videos from online repositories,
aiming to learn joint representations of text, pose, and IMU simultaneously. By
employing video data and contrastive learning, our method seeks to enhance
wearable HAR performance, especially in recognizing subtle activities.Our
experimental findings validate the effectiveness of our approach in improving
HAR performance with IMU data. We demonstrate that models trained with
synthetic IMU data generated from videos using our method surpass existing
approaches in recognizing fine-grained activities.

由于 HAR 中标记传感器数据的稀缺性，先前的研究已经转向使用视频数据合成惯性测量单元（IMU）数据，利用其丰富的活动注释。然而，在真实环境中从视频生成 IMU 数据对 HAR 提出了挑战，归因于合成 IMU 数据的质量较差且在细微、细粒度动作方面的效果有限。本文提出了我们的新颖多模态、多任务和对比基于框架方法 Multi$^3$Net 来解决数据有限的问题。我们的预训练过程使用在线存储库的视频，旨在同时学习文本、姿势和 IMU 的联合表示。通过使用视频数据和对比学习，我们的方法旨在增强可穿戴 HAR 性能，特别是在识别细微活动方面。我们的实验结果验证了我们的方法在利用 IMU 数据改善 HAR 性能方面的有效性。我们证明，使用我们的方法从视频生成的合成 IMU 数据训练的模型在识别细粒度活动方面超越了现有方法。