Training-free Vision Transformer (ViT) architecture search is presented to
search for a better ViT with zero-cost proxies. While ViTs achieve significant
distillation gains from CNN teacher models on small datasets, the current
zero-cost proxies in ViTs do not generalize well to the distillation training
paradigm according to our experimental observations. In this paper, for the
first time, we investigate how to search in a training-free manner with the
help of teacher models and devise an effective Training-free ViT (TVT) search
framework. Firstly, we observe that the similarity of attention maps between
ViT and ConvNet teachers affects distill accuracy notably. Thus, we present a
teacher-aware metric conditioned on the feature attention relations between
teacher and student. Additionally, TVT employs the L2-Norm of the student's
weights as the student-capability metric to improve ranking consistency.
Finally, TVT searches for the best ViT for distilling with ConvNet teachers via
our teacher-aware metric and student-capability metric, resulting in impressive
gains in efficiency and effectiveness. Extensive experiments on various tiny
datasets and search spaces show that our TVT outperforms state-of-the-art
training-free search methods. The code will be released.

本文提出了一种无需训练的视觉 Transformer（ViT）架构搜索方法，通过使用教师模型，利用具有零成本代理的方式搜索更好的 ViT，并在各种小型数据集和搜索空间上展开了大量实验，证明了该方法优于现有的无训练搜索方法。

TVT: 在小数据集上无需训练的视觉 Transformer 搜索

TVT: Training-Free Vision Transformer Search on Tiny Datasets

We prove explicit, i.e. non-asymptotic, error bounds for Markov chain Monte
Carlo methods. The problem is to compute the expectation of a function f with
respect to a measure {\pi}. Different convergence properties of Markov chains
imply different error bounds. For uniformly ergodic and reversible Markov
chains we prove a lower and an upper error bound with respect to the L2 -norm
of f . If there exists an L2 -spectral gap, which is a weaker convergence
property than uniform ergodicity, then we show an upper error bound with
respect to the Lp -norm of f for p > 2. Usually a burn-in period is an
efficient way to tune the algorithm. We provide and justify a recipe how to
choose the burn-in period. The error bounds are applied to the problem of the
integration with respect to a possibly unnormalized density. More precise, we
consider the integration with respect to log-concave densities and the
integration over convex bodies. By the use of the Metropolis algorithm based on
a ball walk and the hit-and-run algorithm it is shown that both problems are
polynomial tractable.

本文研究了马尔科夫链蒙特卡罗方法的误差边界问题，针对不同类型的马尔科夫链给出了相应的上下界，同时提供了选择转化期的配方，并将误差边界应用于积分和概率计算。