It is widely acknowledged that large and sparse models have higher accuracy
than small and dense models under the same model size constraints. This
motivates us to train a large model and then remove its redundant neurons or
weights by pruning. Most existing works pruned the networks in a deterministic
way, the performance of which solely depends on a single pruning criterion and
thus lacks variety. Instead, in this paper, we propose a model pruning strategy
that first generates several pruning masks in a designed random way.
Subsequently, along with an effective mask-selection rule, the optimal mask is
chosen from the pool of mask candidates. To further enhance efficiency, we
introduce an early mask evaluation strategy, mitigating the overhead associated
with training multiple masks. Our extensive experiments demonstrate that this
approach achieves state-of-the-art performance across eight datasets from GLUE,
particularly excelling at high levels of sparsity.

在同等模型大小约束下，大而稀疏的模型普遍比小而密集的模型具有更高的准确性。因此，本文提出了一种模型修剪策略，首先以设计的随机方式生成多个修剪掩码，然后通过一个有效的掩码选择规则从候选掩码组中选择最优掩码。为了进一步提高效率，引入了一种早期掩码评估策略，减少了训练多个掩码的开销。在 GLUE 数据集的八个数据集上进行了广泛的实验证明，该方法在高稀疏水平下实现了最先进的性能。