Neural parameter allocation search (NPAS) automates parameter sharing by
obtaining weights for a network given an arbitrary, fixed parameter budget.
Prior work has two major drawbacks we aim to address. First, there is a
disconnect in the sharing pattern between the search and training steps, where
weights are warped for layers of different sizes during the search to measure
similarity, but not during training, resulting in reduced performance. To
address this, we generate layer weights by learning to compose sets of
SuperWeights, which represent a group of trainable parameters. These
SuperWeights are created to be large enough so they can be used to represent
any layer in the network, but small enough that they are computationally
efficient. The second drawback we address is the method of measuring similarity
between shared parameters. Whereas prior work compared the weights themselves,
we argue this does not take into account the amount of conflict between the
shared weights. Instead, we use gradient information to identify layers with
shared weights that wish to diverge from each other. We demonstrate that our
SuperWeight Networks consistently boost performance over the state-of-the-art
on the ImageNet and CIFAR datasets in the NPAS setting. We further show that
our approach can generate parameters for many network architectures using the
same set of weights. This enables us to support tasks like efficient ensembling
and anytime prediction, outperforming fully-parameterized ensembles with 17%
fewer parameters.

神经参数分配搜索（NPAS）通过在给定任意固定参数预算的情况下获取网络权重来自动化参数共享。我们改进了先前工作中的两个主要缺点，首先是在搜索和训练步骤之间共享模式的不一致性，其在搜索期间对不同大小的层进行权重变形以衡量相似性，但在训练期间不进行变形，从而导致性能降低。其次，我们改进了共享参数之间相似性的测量方法，我们认为之前的方法仅比较权重本身，而未考虑共享权重之间的冲突程度。相反，我们利用梯度信息识别希望在共享权重之间发生分歧的层。我们证明了我们的超重网络在 NPAS 设置中在 ImageNet 和 CIFAR 数据集上始终提升性能。此外，我们还展示了我们的方法可以使用相同的权重为许多网络架构生成参数。这使我们能够支持高效集成和任意时间预测等任务，并且相比完全参数化的集成方法，我们的方法可以使用更少 17％的参数取得更好的性能。