There has been much interest in deploying deep learning algorithms on
low-powered devices, including smartphones, drones, and medical sensors.
However, full-scale deep neural networks are often too resource-intensive in
terms of energy and storage. As a result, the bulk part of the machine learning
operation is therefore often carried out on an edge server, where the data is
compressed and transmitted. However, compressing data (such as images) leads to
transmitting information irrelevant to the supervised task. Another popular
approach is to split the deep network between the device and the server while
compressing intermediate features. To date, however, such split computing
strategies have barely outperformed the aforementioned naive data compression
baselines due to their inefficient approaches to feature compression. This
paper adopts ideas from knowledge distillation and neural image compression to
compress intermediate feature representations more efficiently. Our supervised
compression approach uses a teacher model and a student model with a stochastic
bottleneck and learnable prior for entropy coding (Entropic Student). We
compare our approach to various neural image and feature compression baselines
in three vision tasks and found that it achieves better supervised
rate-distortion performance while maintaining smaller end-to-end latency. We
furthermore show that the learned feature representations can be tuned to serve
multiple downstream tasks.

本研究旨在通过利用知识蒸馏和神经图像压缩的思想，采用先进的监督压缩方法，使用带有随机瓶颈的教师模型和学生模型，以及可学习的熵编码先验方法来更高效地压缩深度学习的中间特征表示。我们在三个视觉任务上将我们的方法与各种神经图像和特征压缩基线进行比较，并发现我们的方法在保持端到端时延更小的情况下，实现了更好的受监督的率失真性能。我们还展示了学习到的特征表示可以调整为服务于多个下游任务。