Traditional initialisation methods, e.g. He and Xavier, have been effective
in avoiding the problem of vanishing or exploding gradients in neural networks.
However, they only use simple pointwise distributions, which model
one-dimensional variables. Moreover, they ignore most information about the
architecture and disregard past training experiences. These limitations can be
overcome by employing generative models for initialisation. In this paper, we
introduce two groups of new initialisation methods. First, we locally
initialise weight groups by employing variational autoencoders. Secondly, we
globally initialise full weight sets by employing graph hypernetworks. We
thoroughly evaluate the impact of the employed generative models on
state-of-the-art neural networks in terms of accuracy, convergence speed and
ensembling. Our results show that global initialisations result in higher
accuracy and faster initial convergence speed. However, the implementation
through graph hypernetworks leads to diminished ensemble performance on out of
distribution data. To counteract, we propose a modification called noise graph
hypernetwork, which encourages diversity in the produced ensemble members.
Furthermore, our approach might be able to transfer learned knowledge to
different image distributions. Our work provides insights into the potential,
the trade-offs and possible modifications of these new initialisation methods.

我们介绍了两组新的初始化方法：第一组是通过应用变分自动编码器来局部初始化权重组，第二组是通过应用图形超网络来全局初始化完整的权重集合。我们通过在精度、收敛速度和集成方面对采用的生成模型对最先进的神经网络进行了彻底的评估。结果表明，全局初始化导致更高的准确性和更快的初始收敛速度，然而通过图形超网络实现的方式会降低对于超出分布数据的集成性能。为了弥补这个问题，我们提出了一种叫噪声图形超网络的修改方法，鼓励在生成的集成成员中产生多样性。此外，我们的方法可能能够将学到的知识传递给不同的图像分布。我们的工作提供了对于这些新的初始化方法的潜力、权衡和可能修改的见解。