In this study, we investigate the task of data pre-selection, which aims to
select instances for labeling from an unlabeled dataset through a single pass,
thereby optimizing performance for undefined downstream tasks with a limited
annotation budget. Previous approaches to data pre-selection relied solely on
visual features extracted from foundation models, such as CLIP and BLIP-2, but
largely ignored the powerfulness of text features. In this work, we argue that,
with proper design, the joint feature space of both vision and text can yield a
better representation for data pre-selection. To this end, we introduce UP-DP,
a simple yet effective unsupervised prompt learning approach that adapts
vision-language models, like BLIP-2, for data pre-selection. Specifically, with
the BLIP-2 parameters frozen, we train text prompts to extract the joint
features with improved representation, ensuring a diverse cluster structure
that covers the entire dataset. We extensively compare our method with the
state-of-the-art using seven benchmark datasets in different settings,
achieving up to a performance gain of 20%. Interestingly, the prompts learned
from one dataset demonstrate significant generalizability and can be applied
directly to enhance the feature extraction of BLIP-2 from other datasets. To
the best of our knowledge, UP-DP is the first work to incorporate unsupervised
prompt learning in a vision-language model for data pre-selection.

本研究通过一次遍历未标记数据集来为有限的标注预算优化性能，从而研究数据预选任务。我们引入了 UP-DP，一种简单而有效的无监督提示学习方法，通过适应 BLIP-2 等视觉语言模型进行数据预选，获得了改进表示的联合特征。在七个基准数据集的不同设置下，我们广泛比较了我们的方法与最先进的方法，取得了 20% 的性能提升。据我们所知，UP-DP 是首个将无监督提示学习引入视觉语言模型中进行数据预选的工作。