We propose expanding the shared Transformer module to produce and initialize
Transformers with diverse depths, enabling adaptation to dynamic resource
constraints. Drawing an analogy to genetic expansibility, we term such module
as learngene. To identify the expansion mechanism, we delve into the
relationship between the layer position and its corresponding weight value, and
find that linear function appropriately approximates this relationship.
Building on this insight, we present Transformer as Linear Expansion of
learnGene (TLEG), a novel approach for flexibly producing and initializing
Transformers of diverse depths. Specifically, to learn learngene, we firstly
construct an auxiliary Transformer linearly expanded from learngene, after
which we train it through employing soft distillation. Subsequently, we can
produce and initialize Transformers of varying depths via linearly expanding
the well-trained learngene, thereby supporting diverse downstream scenarios.
Extensive experiments on ImageNet-1K classification demonstrate that TLEG
achieves comparable or better performance compared to many individual models
trained from scratch, while reducing around 2$\times$ training cost. When
transferring one model to several downstream classification datasets, TLEG
surpasses existing initialization methods by a large margin (e.g., +6.87% on
iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to
produce models of different scales adapting for different resource constraints,
TLEG achieves comparable results while reducing around 19$\times$ parameters
stored to initialize these models and around 5$\times$ training costs, in
contrast to the pre-training and fine-tuning approach.

扩展共享 Transformer 模块以生成和初始化具有不同深度的 Transformer，以适应动态资源约束。