The sparsity of Deep Neural Networks is well investigated to maximize the
performance and reduce the size of overparameterized networks as possible.
Existing methods focus on pruning parameters in the training process by using
thresholds and metrics. Meanwhile, feature similarity between different layers
has not been discussed sufficiently before, which could be rigorously proved to
be highly correlated to the network sparsity in this paper. Inspired by
interlayer feature similarity in overparameterized models, we investigate the
intrinsic link between network sparsity and interlayer feature similarity.
Specifically, we prove that reducing interlayer feature similarity based on
Centered Kernel Alignment (CKA) improves the sparsity of the network by using
information bottleneck theory. Applying such theory, we propose a plug-and-play
CKA-based Sparsity Regularization for sparse network training, dubbed CKA-SR,
which utilizes CKA to reduce feature similarity between layers and increase
network sparsity. In other words, layers of our sparse network tend to have
their own identity compared to each other. Experimentally, we plug the proposed
CKA-SR into the training process of sparse network training methods and find
that CKA-SR consistently improves the performance of several State-Of-The-Art
sparse training methods, especially at extremely high sparsity. Code is
included in the supplementary materials.

本文通过探究网络层的特征相似性与网络稀疏性之间的内在联系，提出了一种基于居中核对齐的稀疏正则化方法（CKA-SR），该方法利用 CKA 降低网络层之间的特征相似度，增加网络稀疏性，并在多种稀疏训练方法上取得了稳定的效果提升，在极高稀疏度下表现尤为明显。

使用恒等层学习稀疏神经网络

Learning Sparse Neural Networks with Identity Layers

Fully exploiting the learning capacity of neural networks requires
overparameterized dense networks. On the other side, directly training sparse
neural networks typically results in unsatisfactory performance. Lottery Ticket
Hypothesis (LTH) provides a novel view to investigate sparse network training
and maintain its capacity. Concretely, it claims there exist winning tickets
from a randomly initialized network found by iterative magnitude pruning and
preserving promising trainability (or we say being in trainable condition). In
this work, we regard the winning ticket from LTH as the subnetwork which is in
trainable condition and its performance as our benchmark, then go from a
complementary direction to articulate the Dual Lottery Ticket Hypothesis
(DLTH): Randomly selected subnetworks from a randomly initialized dense network
can be transformed into a trainable condition and achieve admirable performance
compared with LTH -- random tickets in a given lottery pool can be transformed
into winning tickets. Specifically, by using uniform-randomly selected
subnetworks to represent the general cases, we propose a simple sparse network
training strategy, Random Sparse Network Transformation (RST), to substantiate
our DLTH. Concretely, we introduce a regularization term to borrow learning
capacity and realize information extrusion from the weights which will be
masked. After finishing the transformation for the randomly selected
subnetworks, we conduct the regular finetuning to evaluate the model using fair
comparisons with LTH and other strong baselines. Extensive experiments on
several public datasets and comparisons with competitive approaches validate
our DLTH as well as the effectiveness of the proposed model RST. Our work is
expected to pave a way for inspiring new research directions of sparse network
training in the future. Our code is available at
this https URL

本论文提出 Dual Lottery Ticket Hypothesis 和 Random Sparse Network Transformation 实现稀疏神经网络训练，并通过实验证明了其有效性。