Tensor decompositions have been successfully applied to compress neural
networks. The compression algorithms using tensor decompositions commonly
minimize the approximation error on the weights. Recent work assumes the
approximation error on the weights is a proxy for the performance of the model
to compress multiple layers and fine-tune the compressed model. Surprisingly,
little research has systematically evaluated which approximation errors can be
used to make choices regarding the layer, tensor decomposition method, and
level of compression. To close this gap, we perform an experimental study to
test if this assumption holds across different layers and types of
decompositions, and what the effect of fine-tuning is. We include the
approximation error on the features resulting from a compressed layer in our
analysis to test if this provides a better proxy, as it explicitly takes the
data into account. We find the approximation error on the weights has a
positive correlation with the performance error, before as well as after
fine-tuning. Basing the approximation error on the features does not improve
the correlation significantly. While scaling the approximation error commonly
is used to account for the different sizes of layers, the average correlation
across layers is smaller than across all choices (i.e. layers, decompositions,
and level of compression) before fine-tuning. When calculating the correlation
across the different decompositions, the average rank correlation is larger
than across all choices. This means multiple decompositions can be considered
for compression and the approximation error can be used to choose between them.

通过实验研究表明，神经网络张量分解压缩中，权重近似误差与性能误差之间存在正相关性，而基于特征的近似误差并不能明显提高相关性，可以尝试使用不同的分解方法来进行选择和压缩，使用权重近似误差作为指标。