Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which suggests deeper DNN layers compress representations and hinder OOD performance. Contrary to earlier work, we find the tunnel effect is not universal. Based on 10,584 linear probes, we study the conditions that mitigate the tunnel effect by varying DNN architecture, training dataset, image resolution, and augmentations. We quantify each variable's impact using a novel SHAP analysis. Our results emphasize the danger of generalizing findings from toy datasets to broader contexts.

预训练深度神经网络嵌入广泛使用，然而在下游任务中它们的有效性各不相同。本文通过隧道效应假设研究了影响预训练深度神经网络嵌入进行域外泛化的因素。通过改变网络架构、训练数据集、图像分辨率和增强等条件来研究如何减轻隧道效应，并使用新颖的SHAP分析量化了每个变量的影响。我们的结果强调了将玩具数据集的发现推广到更广泛的背景下的危险性。

预训练模型的外部分布泛化受哪些变量影响？