Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some
pre-trained networks for quickly building numerous networks with different
complexity and performance trade-offs. In this way, the burdens of designing or
training the variable-sized networks, which can be used in application
scenarios with diverse resource constraints, are alleviated. However, SN-Net
still faces a few challenges. 1) Stitching from multiple independently
pre-trained anchors introduces high storage resource consumption. 2) SN-Net
faces challenges to build smaller models for low resource constraints. 3).
SN-Net uses an unlearned initialization method for stitch layers, limiting the
final performance. To overcome these challenges, motivated by the recently
proposed Learngene framework, we propose a novel method called Learngene Pool.
Briefly, Learngene distills the critical knowledge from a large pre-trained
model into a small part (termed as learngene) and then expands this small part
into a few variable-sized models. In our proposed method, we distill one
pretrained large model into multiple small models whose network blocks are used
as learngene instances to construct the learngene pool. Since only one large
model is used, we do not need to store more large models as SN-Net and after
distilling, smaller learngene instances can be created to build small models to
satisfy low resource constraints. We also insert learnable transformation
matrices between the instances to stitch them into variable-sized models to
improve the performance of these models. Exhaustive experiments have been
implemented and the results validate the effectiveness of the proposed
Learngene Pool compared with SN-Net.

通过 Learngene Pool 方法，将 Stitchable Neural Networks (SN-Net) 中存在的挑战解决，从而在低资源约束条件下构建小型和可变尺寸的模型，进一步提升性能。