We present IMU2CLIP, a novel pre-training approach to align Inertial
Measurement Unit (IMU) motion sensor recordings with video and text, by
projecting them into the joint representation space of Contrastive
Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to
translate human motions (as measured by IMU sensors) into their corresponding
textual descriptions and videos -- while preserving the transitivity across
these modalities.
We explore several new IMU-based applications that IMU2CLIP enables, such as
motion-based media retrieval and natural language reasoning tasks with motion
data. In addition, we show that IMU2CLIP can significantly improve the
downstream performance when fine-tuned for each application (e.g. activity
recognition), demonstrating the universal usage of IMU2CLIP as a new
pre-trained resource. Our code will be made publicly available.

IMU2CLIP 是一种新型的预训练方法，用于将惯性测量单元（IMU）运动传感器记录与视频和文本对齐，从而在对比度语言 - 图像预训练 (CLIP) 的联合表示空间中将其投影，该方法允许 IMU2CLIP 将人体动作转化为相应的文本描述和视频，并保持这些模态之间的传递性，用于多种应用，包括基于动作的媒体检索和自然语言推理任务以及活动识别等，并展示了 IMU2CLIP 在每种应用的定制预训练可大大提高下游性能，彰显了其作为预训练资源的通用用途。