Training neural networks requires increasing amounts of memory. Parameter
sharing can reduce memory and communication costs, but existing methods assume
networks have many identical layers and utilize hand-crafted sharing strategies
that fail to generalize. We introduce Neural Parameter Allocation Search
(NPAS), a novel task where the goal is to train a neural network given an
arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which
produce compact networks, as well as a novel high-budget regime, where
additional capacity can be added to boost performance without increasing
inference FLOPs. To address NPAS, we introduce Shapeshifter Networks (SSNs),
which automatically learn where and how to share parameters in a network to
support any parameter budget without requiring any changes to the architecture
or loss function. NPAS and SSNs provide a complete framework for addressing
generalized parameter sharing, and can also be combined with prior work for
additional performance gains. We demonstrate the effectiveness of our approach
using nine network architectures across four diverse tasks, including ImageNet
classification and transformers.

本文提出神经参数分配搜索 (NPAS) 以及 Shapeshifter Networks (SSNs) 算法，通过自动学习神经网络中参数的共享，从而达到内存优化和性能优化的目的。这一方法在多项任务和网络结构中取得良好的效果。