Recently, multimodal prompting, which introduces learnable missing-aware
prompts for all missing modality cases, has exhibited impressive performance.
However, it encounters two critical issues: 1) The number of prompts grows
exponentially as the number of modalities increases; and 2) It lacks robustness
in scenarios with different missing modality settings between training and
inference. In this paper, we propose a simple yet effective prompt design to
address these challenges. Instead of using missing-aware prompts, we utilize
prompts as modality-specific tokens, enabling them to capture the unique
characteristics of each modality. Furthermore, our prompt design leverages
orthogonality between prompts as a key element to learn distinct information
across different modalities and promote diversity in the learned
representations. Extensive experiments demonstrate that our prompt design
enhances both performance and robustness while reducing the number of prompts.

使用具体的提示设计来提高多模态学习的性能和鲁棒性，并减少提示的数量。

朝向健壮的多模式提示：缺失模态

Towards Robust Multimodal Prompting With Missing Modalities

Fine-grained visual classification (FGVC) involves categorizing fine
subdivisions within a broader category, which poses challenges due to subtle
inter-class discrepancies and large intra-class variations. However, prevailing
approaches primarily focus on uni-modal visual concepts. Recent advancements in
pre-trained vision-language models have demonstrated remarkable performance in
various high-level vision tasks, yet the applicability of such models to FGVC
tasks remains uncertain. In this paper, we aim to fully exploit the
capabilities of cross-modal description to tackle FGVC tasks and propose a
novel multimodal prompting solution, denoted as MP-FGVC, based on the
contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a
multimodal prompts scheme and a multimodal adaptation scheme. The former
includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text
Prompt (DaTP), which explicitly highlights the subcategory-specific
discrepancies from the perspectives of both vision and language. The latter
aligns the vision and text prompting elements in a common semantic space,
facilitating cross-modal collaborative reasoning through a Vision-Language
Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a
two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained
CLIP model and expedite efficient adaptation for FGVC. Extensive experiments
conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

本文介绍了一种基于 CLIP 模型的多模态提示解决方案（MP-FGVC），通过跨模态描述充分利用 CLIP 模型在细粒度视觉分类（FGVC）任务中的能力，并且通过多模态的协同推理进一步提高 FGVC 的性能。