The expansion of neural network sizes and the enhancement of image resolution
through modern camera sensors result in heightened memory and power demands for
neural networks. Reducing peak memory, which is the maximum memory consumed
during the execution of a neural network, is critical to deploy neural networks
on edge devices with limited memory budget. A naive approach to reducing peak
memory is aggressive down-sampling of feature maps via pooling with large
stride, which often results in unacceptable degradation in network performance.
To mitigate this problem, we propose residual encoded distillation (ReDistill)
for peak memory reduction in a teacher-student framework, in which a student
network with less memory is derived from the teacher network using aggressive
pooling. We apply our distillation method to multiple problems in computer
vision including image classification and diffusion based image generation. For
image classification, our method yields 2x-3.2x measured peak memory on an edge
GPU with negligible degradation in accuracy for most CNN based architectures.
Additionally, our method yields improved test accuracy for tiny vision
transformer (ViT) based models distilled from large CNN based teacher
architectures. For diffusion-based image generation, our proposed distillation
method yields a denoising network with 4x lower theoretical peak memory while
maintaining decent diversity and fidelity for image generation. Experiments
demonstrate our method's superior performance compared to other feature-based
and response-based distillation methods.

通过对神经网络进行峰值内存降低，我们提出了残差编码蒸馏（ReDistill）技术来在边缘设备上部署具有有限内存预算的神经网络。我们的方法不仅适用于图像分类问题，而且在基于扩散的图像生成中也有出色表现。