Probes are small networks that predict properties of underlying data from
embeddings, and they provide a targeted, effective way to illuminate the
information contained in embeddings. While analysis through the use of probes
has become standard in NLP, there has been much less exploration in vision.
Image foundation models have primarily been evaluated for semantic content.
Better understanding the non-semantic information in popular embeddings (e.g.,
MAE, SimCLR, or CLIP) will shed new light both on the training algorithms and
on the uses for these foundation models. We design a systematic transformation
prediction task and measure the visual content of embeddings along many axes,
including image style, quality, and a range of natural and artificial
transformations. Surprisingly, six embeddings (including SimCLR) encode enough
non-semantic information to identify dozens of transformations. We also
consider a generalization task, where we group similar transformations and hold
out several for testing. We find that image-text models (CLIP and ALIGN) are
better at recognizing new examples of style transfer than masking-based models
(CAN and MAE). Overall, our results suggest that the choice of pre-training
algorithm impacts the types of information in the embedding, and certain models
are better than others for non-semantic downstream tasks.

通过对多种视觉 embedding 进行系统转换任务的测试与比较，发现不同的网络预训练算法会在 embedding 中编码不同类型的图像非语义信息，并且这些信息可以被有效地利用到一些特定的下游应用中。