Traditional initialisation methods, e.g. He and Xavier, have been effective
in avoiding the problem of vanishing or exploding gradients in neural networks.
However, they only use simple pointwise distributions, which model
one-dimensional variables. Moreover, they ignore most information about the
architecture and disregard past training experiences. These limitations can be
overcome by employing generative models for initialisation. In this paper, we
introduce two groups of new initialisation methods. First, we locally
initialise weight groups by employing variational autoencoders. Secondly, we
globally initialise full weight sets by employing graph hypernetworks. We
thoroughly evaluate the impact of the employed generative models on
state-of-the-art neural networks in terms of accuracy, convergence speed and
ensembling. Our results show that global initialisations result in higher
accuracy and faster initial convergence speed. However, the implementation
through graph hypernetworks leads to diminished ensemble performance on out of
distribution data. To counteract, we propose a modification called noise graph
hypernetwork, which encourages diversity in the produced ensemble members.
Furthermore, our approach might be able to transfer learned knowledge to
different image distributions. Our work provides insights into the potential,
the trade-offs and possible modifications of these new initialisation methods.

我们介绍了两组新的初始化方法：第一组是通过应用变分自动编码器来局部初始化权重组，第二组是通过应用图形超网络来全局初始化完整的权重集合。我们通过在精度、收敛速度和集成方面对采用的生成模型对最先进的神经网络进行了彻底的评估。结果表明，全局初始化导致更高的准确性和更快的初始收敛速度，然而通过图形超网络实现的方式会降低对于超出分布数据的集成性能。为了弥补这个问题，我们提出了一种叫噪声图形超网络的修改方法，鼓励在生成的集成成员中产生多样性。此外，我们的方法可能能够将学到的知识传递给不同的图像分布。我们的工作提供了对于这些新的初始化方法的潜力、权衡和可能修改的见解。

从点到源：用生成模型初始化神经网络

From Pointwise to Powerhouse: Initialising Neural Networks with  Generative Models

Neural Architecture Search (NAS) yields state-of-the-art neural networks that
outperform their best manually-designed counterparts. However, previous NAS
methods search for architectures under one set of training hyper-parameters
(i.e., a training recipe), overlooking superior architecture-recipe
combinations. To address this, we present Neural Architecture-Recipe Search
(NARS) to search both (a) architectures and (b) their corresponding training
recipes, simultaneously. NARS utilizes an accuracy predictor that scores
architecture and training recipes jointly, guiding both sample selection and
ranking. Furthermore, to compensate for the enlarged search space, we leverage
"free" architecture statistics (e.g., FLOP count) to pretrain the predictor,
significantly improving its sample efficiency and prediction reliability. After
training the predictor via constrained iterative optimization, we run fast
evolutionary searches in just CPU minutes to generate architecture-recipe pairs
for a variety of resource constraints, called FBNetV3. FBNetV3 makes up a
family of state-of-the-art compact neural networks that outperform both
automatically and manually-designed competitors. For example, FBNetV3 matches
both EfficientNet and ResNeSt accuracy on ImageNet with up to 2.0x and 7.1x
fewer FLOPs, respectively. Furthermore, FBNetV3 yields significant performance
gains for downstream object detection tasks, improving mAP despite 18% fewer
FLOPs and 34% fewer parameters than EfficientNet-based equivalents.

提出了一种同时搜索神经架构和相应的训练参数的新方法 —— 神经架构及训练参数搜索（NARS），其中利用了免费的架构统计信息（如 FLOP 计数）来预先训练准确性预测器，使用受约束的迭代优化方法训练预测器，然后快速进行进化搜索来生成一系列精简的神经网络，称为 FBNetV3，它在资源限制条件下的准确率优于其他自动或手动设计的对手。