In energy-efficient schemes, finding the optimal size of deep learning models
is very important and has a broad impact. Meanwhile, recent studies have
reported an unexpected phenomenon, the sparse double descent: as the model's
sparsity increases, the performance first worsens, then improves, and finally
deteriorates. Such a non-monotonic behavior raises serious questions about the
optimal model's size to maintain high performance: the model needs to be
sufficiently over-parametrized, but having too many parameters wastes training
resources.
In this paper, we aim to find the best trade-off efficiently. More precisely,
we tackle the occurrence of the sparse double descent and present some
solutions to avoid it. Firstly, we show that a simple $\ell_2$ regularization
method can help to mitigate this phenomenon but sacrifices the
performance/sparsity compromise. To overcome this problem, we then introduce a
learning scheme in which distilling knowledge regularizes the student model.
Supported by experimental results achieved using typical image classification
setups, we show that this approach leads to the avoidance of such a phenomenon.

在能效方案中，找到深度学习模型的最佳大小非常重要且具有广泛的影响。本文的研究目标是高效地寻找最佳折中方法，针对稀疏双下降现象提出了解决方案，包括使用 L2 正则化方法以及通过知识蒸馏来规范学生模型，实验结果表明这些方法可以避免该现象的发生。