Cross-modality transfer aims to leverage large pretrained models to complete
tasks that may not belong to the modality of pretraining data. Existing works
achieve certain success in extending classical finetuning to cross-modal
scenarios, yet we still lack understanding about the influence of modality gap
on the transfer. In this work, a series of experiments focusing on the source
representation quality during transfer are conducted, revealing the connection
between larger modality gap and lesser knowledge reuse which means ineffective
transfer. We then formalize the gap as the knowledge misalignment between
modalities using conditional distribution P(Y|X). Towards this problem, we
present Modality kNowledge Alignment (MoNA), a meta-learning approach that
learns target data transformation to reduce the modality knowledge discrepancy
ahead of the transfer. Experiments show that out method enables better reuse of
source modality knowledge in cross-modality transfer, which leads to
improvements upon existing finetuning methods.

通过进行一系列实验，本研究揭示了模态间差距与知识重用程度之间的关联，定义了作为模态知识差异的知识不匹配性，并提出了模态知识对齐元学习方法（MoNA）来减小模态间的知识差异，实验证明该方法可以在跨模态迁移中更好地重用源模态知识，并改进现有的微调方法。

跨模态迁移的学习模态知识对齐

Learning Modality Knowledge Alignment for Cross-Modality Transfer

The development of robust, generalized models in human activity recognition
(HAR) has been hindered by the scarcity of large-scale, labeled data sets.
Recent work has shown that virtual IMU data extracted from videos using
computer vision techniques can lead to substantial performance improvements
when training HAR models combined with small portions of real IMU data.
Inspired by recent advances in motion synthesis from textual descriptions and
connecting Large Language Models (LLMs) to various AI models, we introduce an
automated pipeline that first uses ChatGPT to generate diverse textual
descriptions of activities. These textual descriptions are then used to
generate 3D human motion sequences via a motion synthesis model, T2M-GPT, and
later converted to streams of virtual IMU data. We benchmarked our approach on
three HAR datasets (RealWorld, PAMAP2, and USC-HAD) and demonstrate that the
use of virtual IMU training data generated using our new approach leads to
significantly improved HAR model performance compared to only using real IMU
data. Our approach contributes to the growing field of cross-modality transfer
methods and illustrate how HAR models can be improved through the generation of
virtual training data that do not require any manual effort.

通过聊天 GPT 生成不同的文本描述活动的文字描述，通过运动合成模型 T2M-GPT 生成 3D 人体运动序列，然后转换成流的模拟 IMU 数据，将其与部分真实 IMU 数据结合训练 HAR 模型可以显著提高其性能