Embeddings produced by pre-trained deep neural networks (DNNs) are widely
used; however, their efficacy for downstream tasks can vary widely. We study
the factors influencing out-of-distribution (OOD) generalization of pre-trained
DNN embeddings through the lens of the tunnel effect hypothesis, which suggests
deeper DNN layers compress representations and hinder OOD performance. Contrary
to earlier work, we find the tunnel effect is not universal. Based on 10,584
linear probes, we study the conditions that mitigate the tunnel effect by
varying DNN architecture, training dataset, image resolution, and
augmentations. We quantify each variable's impact using a novel SHAP analysis.
Our results emphasize the danger of generalizing findings from toy datasets to
broader contexts.

预训练深度神经网络嵌入广泛使用，然而在下游任务中它们的有效性各不相同。本文通过隧道效应假设研究了影响预训练深度神经网络嵌入进行域外泛化的因素。通过改变网络架构、训练数据集、图像分辨率和增强等条件来研究如何减轻隧道效应，并使用新颖的 SHAP 分析量化了每个变量的影响。我们的结果强调了将玩具数据集的发现推广到更广泛的背景下的危险性。