Recent works have explored the use of weight sparsity to improve the training
efficiency (test accuracy w.r.t training FLOPs) of deep neural networks (DNNs).
These works aim to reduce training FLOPs but training with sparse weights often
leads to accuracy loss or requires longer training schedules, making the
resulting training efficiency less clear. In contrast, we focus on using
sparsity to increase accuracy while using the same FLOPs as the dense model and
show training efficiency gains through higher accuracy. In this work, we
introduce Sparse-IFT, a family of Sparse Iso-FLOP Transformations which are
used as drop-in replacements for dense layers to improve their representational
capacity and FLOP efficiency. Each transformation is parameterized by a single
hyperparameter (sparsity level) and provides a larger search space to find
optimal sparse masks. Without changing any training hyperparameters, replacing
dense layers with Sparse-IFT leads to significant improvements across computer
vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on
ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL), both matching
larger dense model variants that use 2x or more FLOPs. To our knowledge, this
is the first work to demonstrate the use of sparsity for improving the accuracy
of dense models via a simple-to-use set of sparse transformations. Code is
available at: this https URL.

通过引入一种基于单一超参数的稀疏 Iso-FLOP 变换族，本文试图使用稀疏性提高密集模型的准确性和 FLOP 效率，实现不更改任何训练超参数而在计算机视觉和自然语言处理任务中获得显著的提高，例如 ResNet-18 在 ImageNet 上提高了 3.5％，GPT-3 Small on WikiText-103 下降了 0.4 个 PPL，并在不增加 FLOPs 的前提下，与使用 2 倍或更多 FLOPs 的大型密集模型变体相匹配。