Data-free compression raises a new challenge because the original training
dataset for a pre-trained model to be compressed is not available due to
privacy or transmission issues. Thus, a common approach is to compute a
reconstructed training dataset before compression. The current reconstruction
methods compute the reconstructed training dataset with a generator by
exploiting information from the pre-trained model. However, current
reconstruction methods focus on extracting more information from the
pre-trained model but do not leverage network engineering. This work is the
first to consider network engineering as an approach to design the
reconstruction method. Specifically, we propose the AutoReCon method, which is
a neural architecture search-based reconstruction method. In the proposed
AutoReCon method, the generator architecture is designed automatically given
the pre-trained model for reconstruction. Experimental results show that using
generators discovered by the AutoRecon method always improve the performance of
data-free compression.

本研究提出了 AutoReCon 方法，这是一种基于神经架构搜索的重构方法，通过自动设计生成器结构以实现对预先训练模型重构的效果达到提高数据无关压缩的方法。

AutoReCon：基于神经网络结构搜索的零数据压缩重建技术

AutoReCon: Neural Architecture Search-based Reconstruction for Data-free  Compression

User data confidentiality protection is becoming a rising challenge in the
present deep learning research. Without access to data, conventional
data-driven model compression faces a higher risk of performance degradation.
Recently, some works propose to generate images from a specific pretrained
model to serve as training data. However, the inversion process only utilizes
biased feature statistics stored in one model and is from low-dimension to
high-dimension. As a consequence, it inevitably encounters the difficulties of
generalizability and inexact inversion, which leads to unsatisfactory
performance. To address these problems, we propose MixMix based on two simple
yet effective techniques: (1) Feature Mixing: utilizes various models to
construct a universal feature space for generalized inversion; (2) Data Mixing:
mixes the synthesized images and labels to generate exact label information. We
prove the effectiveness of MixMix from both theoretical and empirical
perspectives. Extensive experiments show that MixMix outperforms existing
methods on the mainstream compression tasks, including quantization, knowledge
distillation, and pruning. Specifically, MixMix achieves up to 4% and 20%
accuracy uplift on quantization and pruning, respectively, compared to existing
data-free compression work.

本文提出了基于 MixMix 的两个简单而有效的技术来解决深度学习研究中面临的用户数据机密保护和模型压缩的问题，同时证明了理论和实验的有效性，比现有的无数据压缩方法在量化、知识蒸馏和剪枝等主流压缩任务上取得了更好的表现。

MixMix：数据无需压缩的关键在于特征和数据混合

MixMix: All You Need for Data-Free Compression Are Feature and Data  Mixing

We present a computationally efficient method for compressing a trained
neural network without using real data. We break the problem of data-free
network compression into independent layer-wise compressions. We show how to
efficiently generate layer-wise training data using only a pretrained network.
We use this data to perform independent layer-wise compressions on the
pretrained network. We also show how to precondition the network to improve the
accuracy of our layer-wise compression method. We present results for
layer-wise compression using quantization and pruning. When quantizing, we
compress with higher accuracy than related works while using orders of
magnitude less compute. When compressing MobileNetV2 and evaluating on
ImageNet, our method outperforms existing methods for quantization at all
bit-widths, achieving a $+0.34\%$ improvement in $8$-bit quantization, and a
stronger improvement at lower bit-widths (up to a $+28.50\%$ improvement at $5$
bits). When pruning, we outperform baselines of a similar compute envelope,
achieving $1.5$ times the sparsity rate at the same accuracy. We also show how
to combine our efficient method with high-compute generative methods to improve
upon their results.

本文介绍了一种基于预先训练网络的数据自由网络压缩方法，该方法采用分层压缩技术、量化和修剪技术，能够实现高精度、高效率的对神经网络进行压缩，还能与高效的数据生成方法相结合以更进一步提高压缩结果。