Large-scale vision-language (V-L) models have demonstrated remarkable generalization capabilities for downstream tasks through prompt tuning. However, their performance suffers significantly in the presence of class imbalance, a common issue in real-world scenarios. In this paper, we investigate the effects of class imbalance on the generalization performance of V-L models and extend Neural Collapse phenomenon to these models, revealing the geometric reasons behind the impact of class imbalance on their generalization ability. To address this problem, we propose Neural Collapse based Prompt Tuning (NPT), a novel method that optimizes prompts so that both text and image features satisfy the same simplex ETF structure. NPT incorporates two regularization terms, geometric de-biasing and multi-modal isomorphism, to enhance the robustness of V-L models under class imbalance conditions while maintaining their generalization capabilities. Our comprehensive experiments show that NPT outperforms existing prompt learning techniques across 11 diverse image recognition datasets, achieving an absolute average gain of 2.63\% for novel classes and 2.47\% for harmonic mean when facing imbalanced data.

本文通过研究类别不平衡对大规模视觉语言模型的影响，提出了一种基于神经坍塌现象的Prompt Tuning方法(NPT)来增强其鲁棒性，并在11个不同的图像识别数据集上展开了综合实验，结果表明NPT在不平衡数据条件下的表现优于现有的prompt learning技术。

跨越鸿沟：基于神经崩溃启发的提示调整，应对分类不平衡的普适性