Overparameterized neural networks generalize well but are expensive to train.
Ideally, one would like to reduce their computational cost while retaining
their generalization benefits. Sparse model training is a simple and promising
approach to achieve this, but there remain challenges as existing methods
struggle with accuracy loss, slow training runtime, or difficulty in
sparsifying all model components. The core problem is that searching for a
sparsity mask over a discrete set of sparse matrices is difficult and
expensive. To address this, our main insight is to optimize over a continuous
superset of sparse matrices with a fixed structure known as products of
butterfly matrices. As butterfly matrices are not hardware efficient, we
propose simple variants of butterfly (block and flat) to take advantage of
modern hardware. Our method (Pixelated Butterfly) uses a simple fixed sparsity
pattern based on flat block butterfly and low-rank matrices to sparsify most
network layers (e.g., attention, MLP). We empirically validate that Pixelated
Butterfly is 3x faster than butterfly and speeds up training to achieve
favorable accuracy--efficiency tradeoffs. On the ImageNet classification and
WikiText-103 language modeling tasks, our sparse models train up to 2.5x faster
than the dense MLP-Mixer, Vision Transformer, and GPT-2 medium with no drop in
accuracy.

该研究提出了一种名为 Pixelated Butterfly 的稀疏模型训练方法，通过优化连续超集上的蝶式矩阵，采用简单的蝶式块和低秩矩阵稀疏化网络层，使训练速度提高 3 倍，并在 ImageNet 分类和 WikiText-103 语言建模任务中，使稀疏模型的训练速度比密集的 MLP-Mixer、Vision Transformer 和 GPT-2 medium 提高了 2.5 倍，而且精确度并未下降