In this paper we apply a compressibility loss that enables learning highly
compressible neural network weights. The loss was previously proposed as a
measure of negated sparsity of a signal, yet in this paper we show that
minimizing this loss also enforces the non-zero parts of the signal to have
very low entropy, thus making the entire signal more compressible. For an
optimization problem where the goal is to minimize the compressibility loss
(the objective), we prove that at any critical point of the objective, the
weight vector is a ternary signal and the corresponding value of the objective
is the squared root of the number of non-zero elements in the signal, thus
directly related to sparsity. In the experiments, we train neural networks with
the compressibility loss and we show that the proposed method achieves weight
sparsity and compression ratios comparable with the state-of-the-art.

本文介绍一种压缩性损失的方法，使神经网络学习高度压缩的神经网络权重，采用这种方法可以使信号的非零部分具有极低的熵，从而整个信号更容易被压缩，我们证明了目标函数的临界点可以使权重向量成为三元信号，其取决于稀疏度，我们实验了这种方法，结果发现它可以最大限度地保留权重，使神经网络的压缩比与现有的最先进方法相似。