Cross-modality transfer aims to leverage large pretrained models to complete
tasks that may not belong to the modality of pretraining data. Existing works
achieve certain success in extending classical finetuning to cross-modal
scenarios, yet we still lack understanding about the influence of modality gap
on the transfer. In this work, a series of experiments focusing on the source
representation quality during transfer are conducted, revealing the connection
between larger modality gap and lesser knowledge reuse which means ineffective
transfer. We then formalize the gap as the knowledge misalignment between
modalities using conditional distribution P(Y|X). Towards this problem, we
present Modality kNowledge Alignment (MoNA), a meta-learning approach that
learns target data transformation to reduce the modality knowledge discrepancy
ahead of the transfer. Experiments show that out method enables better reuse of
source modality knowledge in cross-modality transfer, which leads to
improvements upon existing finetuning methods.

通过进行一系列实验，本研究揭示了模态间差距与知识重用程度之间的关联，定义了作为模态知识差异的知识不匹配性，并提出了模态知识对齐元学习方法（MoNA）来减小模态间的知识差异，实验证明该方法可以在跨模态迁移中更好地重用源模态知识，并改进现有的微调方法。