Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some
pre-trained networks for quickly building numerous networks with different
complexity and performance trade-offs. In this way, the burdens of designing or
training the variable-sized networks, which can be used in application
scenarios with diverse resource constraints, are alleviated. However, SN-Net
still faces a few challenges. 1) Stitching from multiple independently
pre-trained anchors introduces high storage resource consumption. 2) SN-Net
faces challenges to build smaller models for low resource constraints. 3).
SN-Net uses an unlearned initialization method for stitch layers, limiting the
final performance. To overcome these challenges, motivated by the recently
proposed Learngene framework, we propose a novel method called Learngene Pool.
Briefly, Learngene distills the critical knowledge from a large pre-trained
model into a small part (termed as learngene) and then expands this small part
into a few variable-sized models. In our proposed method, we distill one
pretrained large model into multiple small models whose network blocks are used
as learngene instances to construct the learngene pool. Since only one large
model is used, we do not need to store more large models as SN-Net and after
distilling, smaller learngene instances can be created to build small models to
satisfy low resource constraints. We also insert learnable transformation
matrices between the instances to stitch them into variable-sized models to
improve the performance of these models. Exhaustive experiments have been
implemented and the results validate the effectiveness of the proposed
Learngene Pool compared with SN-Net.

通过 Learngene Pool 方法，将 Stitchable Neural Networks (SN-Net) 中存在的挑战解决，从而在低资源约束条件下构建小型和可变尺寸的模型，进一步提升性能。

通过 Learngene Pool 构建可变尺寸模型

Building Variable-sized Models via Learngene Pool

The public model zoo containing enormous powerful pretrained model families
(e.g., ResNet/DeiT) has reached an unprecedented scope than ever, which
significantly contributes to the success of deep learning. As each model family
consists of pretrained models with diverse scales (e.g., DeiT-Ti/S/B), it
naturally arises a fundamental question of how to efficiently assemble these
readily available models in a family for dynamic accuracy-efficiency trade-offs
at runtime. To this end, we present Stitchable Neural Networks (SN-Net), a
novel scalable and efficient framework for model deployment which cheaply
produces numerous networks with different complexity and performance trade-offs
given a family of pretrained neural networks, which we call anchors.
Specifically, SN-Net splits the anchors across the blocks/layers and then
stitches them together with simple stitching layers to map the activations from
one anchor to another. With only a few epochs of training, SN-Net effectively
interpolates between the performance of anchors with varying scales. At
runtime, SN-Net can instantly adapt to dynamic resource constraints by
switching the stitching positions. Extensive experiments on ImageNet
classification demonstrate that SN-Net can obtain on-par or even better
performance than many individually trained networks while supporting diverse
deployment scenarios. For example, by stitching Swin Transformers, we challenge
hundreds of models in Timm model zoo with a single network. We believe this new
elastic model framework can serve as a strong baseline for further research in
wider communities.

提出了一种 Scalable 并有效的框架来对预训练神经网络进行动态准确率 - 效率权衡的组装，该方案称为 Stitchable Neural Networks（SN-Net）；该方案在 ImageNet 分类任务中取得了与已训练网络相当甚至更好的表现，并支持多种部署场景。