Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous
owing to their high accuracy on real-world tasks. However, this increase in
accuracy comes at the cost of computationally expensive models leading to
higher prediction latencies. Prior efforts to reduce this latency such as
quantization, model distillation, and any-time prediction models typically
trade-off accuracy for performance. In this work, we observe that caching
intermediate layer outputs can help us avoid running all the layers of a DNN
for a sizeable fraction of inference requests. We find that this can
potentially reduce the number of effective layers by half for 91.58% of
CIFAR-10 requests run on ResNet-18. We present Freeze Inference, a system that
introduces approximate caching at each intermediate layer and we discuss
techniques to reduce the cache size and improve the cache hit rate. Finally, we
discuss some of the open research challenges in realizing such a design.

通过缓存中间层输出，可以避免运行所有 DNN 层，从而减少预测延迟，本文介绍了 Freeze Inference 系统，它通过在每个中间层引入逼近缓存，并讨论如何减小缓存大小和提高缓存命中率，未来的挑战也被提出。