Sparse neural networks have shown similar or better generalization
performance than their dense counterparts while having higher parameter
efficiency. This has motivated a number of works to learn, induce, or search
for high performing sparse networks. While reports of quality or efficiency
gains are impressive, standard baselines are lacking, therefore hindering
having reliable comparability and reproducibility across methods. In this work,
we provide an evaluation approach and a naive Random Search baseline method for
finding good sparse configurations. We apply Random Search on the node space of
an overparameterized network with the goal of finding better initialized sparse
sub-networks that are positioned more advantageously in the loss landscape. We
record sparse network post-training performances at various levels of sparsity
and compare against both their fully connected parent networks and random
sparse configurations at the same sparsity levels. We observe that for this
architecture search task, initialized sparse networks found by Random Search
neither perform better nor converge more efficiently than their random
counterparts. Thus we conclude that Random Search may be viewed as a suitable
neutral baseline for sparsity search methods.

稀疏神经网络通过更高的参数效率展现出与密集网络类似甚至更好的泛化性能，但由于缺乏标准基线方法，可靠的比较和可重现性受到限制。本研究提供了一种评估方法和一种基于随机搜索的朴素基线方法，用于寻找良好的稀疏配置，并发现随机搜索在初始化稀疏子网络中并不能表现得更好或更高效。因此，我们得出结论认为随机搜索可以被视为适用于稀疏搜索方法的中性基准线。