Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some
pre-trained networks for quickly building numerous networks with different
complexity and performance trade-offs. In this way, the burdens of designing or
training the variable-sized networks, which can be used in application
scenarios with diverse resource constraints, are alleviated. However, SN-Net
still faces a few challenges. 1) Stitching from multiple independently
pre-trained anchors introduces high storage resource consumption. 2) SN-Net
faces challenges to build smaller models for low resource constraints. 3).
SN-Net uses an unlearned initialization method for stitch layers, limiting the
final performance. To overcome these challenges, motivated by the recently
proposed Learngene framework, we propose a novel method called Learngene Pool.
Briefly, Learngene distills the critical knowledge from a large pre-trained
model into a small part (termed as learngene) and then expands this small part
into a few variable-sized models. In our proposed method, we distill one
pretrained large model into multiple small models whose network blocks are used
as learngene instances to construct the learngene pool. Since only one large
model is used, we do not need to store more large models as SN-Net and after
distilling, smaller learngene instances can be created to build small models to
satisfy low resource constraints. We also insert learnable transformation
matrices between the instances to stitch them into variable-sized models to
improve the performance of these models. Exhaustive experiments have been
implemented and the results validate the effectiveness of the proposed
Learngene Pool compared with SN-Net.

通过 Learngene Pool 方法，将 Stitchable Neural Networks (SN-Net) 中存在的挑战解决，从而在低资源约束条件下构建小型和可变尺寸的模型，进一步提升性能。

通过 Learngene Pool 构建可变尺寸模型

Building Variable-sized Models via Learngene Pool

We propose expanding the shared Transformer module to produce and initialize
Transformers with diverse depths, enabling adaptation to dynamic resource
constraints. Drawing an analogy to genetic expansibility, we term such module
as learngene. To identify the expansion mechanism, we delve into the
relationship between the layer position and its corresponding weight value, and
find that linear function appropriately approximates this relationship.
Building on this insight, we present Transformer as Linear Expansion of
learnGene (TLEG), a novel approach for flexibly producing and initializing
Transformers of diverse depths. Specifically, to learn learngene, we firstly
construct an auxiliary Transformer linearly expanded from learngene, after
which we train it through employing soft distillation. Subsequently, we can
produce and initialize Transformers of varying depths via linearly expanding
the well-trained learngene, thereby supporting diverse downstream scenarios.
Extensive experiments on ImageNet-1K classification demonstrate that TLEG
achieves comparable or better performance compared to many individual models
trained from scratch, while reducing around 2$\times$ training cost. When
transferring one model to several downstream classification datasets, TLEG
surpasses existing initialization methods by a large margin (e.g., +6.87% on
iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to
produce models of different scales adapting for different resource constraints,
TLEG achieves comparable results while reducing around 19$\times$ parameters
stored to initialize these models and around 5$\times$ training costs, in
contrast to the pre-training and fine-tuning approach.

扩展共享 Transformer 模块以生成和初始化具有不同深度的 Transformer，以适应动态资源约束。

作为学习基因的线性扩展的变压器

Transformer as Linear Expansion of Learngene

During the continuous evolution of one organism's ancestry, its genes
accumulate extensive experiences and knowledge, enabling newborn descendants to
rapidly adapt to their specific environments. Motivated by this observation, we
propose a novel machine learning paradigm \textit{Learngene} to enable learning
models to incorporate three key characteristics of genes. (i) Accumulating: the
knowledge is accumulated during the continuous learning of an \textbf{ancestry
model}. (ii) Condensing: the exhaustive accumulated knowledge is condensed into
a much more compact information piece, \ie \textbf{learngene}. (iii):
Inheriting: the condensed \textbf{learngene} is inherited to make it easier for
\textbf{descendant models} to adapt to new environments. Since accumulating has
been studied in some well-developed paradigms like large-scale pre-training and
lifelong learning, we focus on condensing and inheriting, which induces three
key issues and we provide the preliminary solutions to these issues in this
paper: (i) \textit{Learngene} Form: the \textbf{learngene} is set to a few
integral layers that can preserve the most commonality. (ii) \textit{Learngene}
Condensing: we identify which layers among the ancestry model have the most
similarity as one pseudo descendant model. (iii) \textit{Learngene} Inheriting:
to construct distinct descendant models for specific downstream tasks, we stack
some randomly initialized layers to the \textbf{learngene} layers. Extensive
experiments of various settings, including using different network
architectures like Vision Transformer (ViT) and Convolutional Neural Networks
(CNNs) on different datasets, are carried out to confirm five advantages and
two characteristics of \textit{Learngene}.

提出了一种新的机器学习范式 Learngene，其主要特点是能够在一个祖先模型的连续学习期间积累知识，把这些经验压缩成紧凑的信息元素（即 Learngene），并将其遗传给后代模型，使其更容易适应新环境。

Learngene：从祖先模型继承压缩知识到后代模型

Learngene: Inheriting Condensed Knowledge from the Ancestry Model to  Descendant Models

Although deep learning has made significant progress on fixed large-scale
datasets, it typically encounters challenges regarding improperly detecting
unknown/unseen classes in the open-world scenario, over-parametrized, and
overfitting small samples. Since biological systems can overcome the above
difficulties very well, individuals inherit an innate gene from collective
creatures that have evolved over hundreds of millions of years and then learn
new skills through few examples. Inspired by this, we propose a practical
collective-individual paradigm where an evolution (expandable) network is
trained on sequential tasks and then recognize unknown classes in real-world.
Moreover, the learngene, i.e., the gene for learning initialization rules of
the target model, is proposed to inherit the meta-knowledge from the collective
model and reconstruct a lightweight individual model on the target task.
Particularly, a novel criterion is proposed to discover learngene in the
collective model, according to the gradient information. Finally, the
individual model is trained only with few samples on the target learning tasks.
We demonstrate the effectiveness of our approach in an extensive empirical
study and theoretical analysis.

提出了一种实用的集体 - 个体范式，该范式在序列任务上训练一个进化（可扩展）网络，并且通过从集体模型中发现 learngene，重建轻量级个体模型，经过实验研究和理论分析证明，本文的方法在实现 few-shot learning 场景中取得了很好的效果