Large models have demonstrated exceptional generalization capabilities in
computer vision and natural language processing. Recent efforts have focused on
enhancing these models with multimodal processing abilities. However,
addressing the challenges posed by scenarios where one modality is absent
remains a significant hurdle. In response to this issue, we propose a robust
latent representation tuning method for large models. Specifically, our
approach introduces a modality latent translation module to maximize the
correlation between modalities. Following this, a newly designed fusion module
is employed to facilitate information interaction between the modalities. In
this framework, not only are common semantics refined during training, but the
method also yields robust representations in the absence of one modality.
Importantly, our method maintains the frozen state of the image and text
foundation models to preserve their abilities acquired through large-scale
pretraining. We conduct experiments on several public datasets, and the results
underscore the effectiveness of our proposed method.

通过引入模态潜在转换模块和新设计的融合模块，提出了一种强大的大型模型的潜在表示调优方法，以最大化模态之间的相关性，并提供在某一模态缺失情况下的稳健表示，同时保留图像和文本基础模型的冻结状态以保留其大规模预训练所获得的能力。实验证明了该方法的有效性。