Foundation vision-language models have enabled remarkable zero-shot
transferability of the pre-trained representations to a wide range of
downstream tasks. However, to solve a new task, zero-shot transfer still
necessitates human guidance to define visual categories that appear in the
data. Here, we show that fully unsupervised transfer emerges when searching for
the labeling of a dataset that induces maximal margin classifiers in
representation spaces of different foundation models. We present TURTLE, a
fully unsupervised method that effectively employs this guiding principle to
uncover the underlying labeling of a downstream dataset without any supervision
and task-specific representation learning. We evaluate TURTLE on a diverse
benchmark suite of 26 datasets and show that it achieves new state-of-the-art
unsupervised performance. Furthermore, TURTLE, although being fully
unsupervised, outperforms zero-shot transfer baselines on a wide range of
datasets. In particular, TURTLE matches the average performance of CLIP
zero-shot on 26 datasets by employing the same representation space, spanning a
wide range of architectures and model sizes. By guiding the search for the
underlying labeling using the representation spaces of two foundation models,
TURTLE surpasses zero-shot transfer and unsupervised prompt tuning baselines,
demonstrating the surprising power and effectiveness of unsupervised transfer.

基于不同基础模型的表示空间引导搜索，TURTLE 发现了下游数据集中的潜在标签而无需任何监督学习，并在 26 个数据集上取得了新的无监督性能的最新成果，超越了零样例传递和无监督提示调优基线的平均表现。