This paper investigates the Neural Tangent Kernel (NTK) to search vision
transformers without training. In contrast with the previous observation that
NTK-based metrics can effectively predict CNNs performance at initialization,
we empirically show their inefficacy in the ViT search space. We hypothesize
that the fundamental feature learning preference within ViT contributes to the
ineffectiveness of applying NTK to NAS for ViT. We both theoretically and
empirically validate that NTK essentially estimates the ability of neural
networks that learn low-frequency signals, completely ignoring the impact of
high-frequency signals in feature learning. To address this limitation, we
propose a new method called ViNTK that generalizes the standard NTK to the
high-frequency domain by integrating the Fourier features from inputs.
Experiments with multiple ViT search spaces on image classification and
semantic segmentation tasks show that our method can significantly speed up
search costs over prior state-of-the-art NAS for ViT while maintaining similar
performance on searched architectures.

本论文探讨了神经切线核（NTK）在没有训练的情况下搜索视觉变换器。通过与先前观察到的基于 NTK 的指标在初始化时能有效预测 CNN 的性能相比较，我们通过实证表明它们在 ViT 搜索空间中的无效性。我们假设 ViT 中的基本特征学习偏好导致将 NTK 应用于 ViT 的 NAS 的无效性。我们从理论和实证角度验证了 NTK 基本上是估计学习低频信号的神经网络的能力，完全忽略了特征学习中高频信号的影响。为了解决这个局限性，我们提出了一种名为 ViNTK 的新方法，通过将傅里叶特征从输入中整合到高频域中，将标准 NTK 推广到高频领域。在图像分类和语义分割任务的多个 ViT 搜索空间上的实验表明，我们的方法可以在维持类似的性能的同时显著加快 ViT 的搜索成本。