Conventional scaling of neural networks typically involves designing a base network and growing different dimensions like width, depth, etc. of the same by some predefined scaling factors. We introduce an automated scaling approach leveraging second-order loss landscape information. Our method is flexible towards skip connections a mainstay in modern vision transformers. Our training-aware method jointly scales and trains transformers without additional training iterations. Motivated by the hypothesis that not all neurons need uniform depth complexity, our approach embraces depth heterogeneity. Extensive evaluations on DeiT-S with ImageNet100 show a 2.5% accuracy gain and 10% parameter efficiency improvement over conventional scaling. Scaled networks demonstrate superior performance upon training small scale datasets from scratch. We introduce the first intact scaling mechanism for vision transformers, a step towards efficient model scaling.

基于第二阶损失景观信息的自动缩放方法，以灵活适应视觉变换器中的跳跃连接；在DeiT-S与ImageNet100上广泛评估，相较于传统缩放，准确率提高2.5%，参数效率提高10%；缩放网络在从头训练小规模数据集时表现出卓越性能，是视觉变换器的首个完整缩放机制，实现高效模型缩放的一步。

超越均匀缩放：探索神经结构中的深度异质性