Methods that sparsify a network at initialization are important in practice
because they greatly improve the efficiency of both learning and inference. Our
work is based on a recently proposed decomposition of the Neural Tangent Kernel
(NTK) that has decoupled the dynamics of the training process into a
data-dependent component and an architecture-dependent kernel - the latter
referred to as Path Kernel. That work has shown how to design sparse neural
networks for faster convergence, without any training data, using the
Synflow-L2 algorithm. We first show that even though Synflow-L2 is optimal in
terms of convergence, for a given network density, it results in sub-networks
with "bottleneck" (narrow) layers - leading to poor performance as compared to
other data-agnostic methods that use the same number of parameters. Then we
propose a new method to construct sparse networks, without any training data,
referred to as Paths with Higher-Edge Weights (PHEW). PHEW is a probabilistic
network formation method based on biased random walks that only depends on the
initial weights. It has similar path kernel properties as Synflow-L2 but it
generates much wider layers, resulting in better generalization and
performance. PHEW achieves significant improvements over the data-independent
SynFlow and SynFlow-L2 methods at a wide range of network densities.

本文基于神经切比雪夫内核（NTK）的分解，提出了一种基于路径加权的概率网络形成方法，可在不需要训练数据的情况下构建稀疏神经网络，并取得更好的泛化性能。