Advances in deep learning have led to substantial increases in prediction accuracy as well as the cost of rendering predictions. We conjecture that for a majority of real-world inputs, the recent advances in deep learning have created models that effectively "over-think" on simple inputs. In this paper we revisit the classic idea of prediction cascades to reduce prediction costs. We introduce the "I Don't Know" (IDK) prediction cascades framework, a general framework for constructing prediction cascades for arbitrary multi-class prediction tasks. We propose two baseline methods for constructing cascades as well as a new objective within this framework and evaluate these techniques on a range of benchmark and real-world datasets to demonstrate the prediction cascades can achieve 1.7-10.5x speedups in image classification tasks while maintaining comparable accuracy to state-of-the-art models. When combined with human experts, prediction cascades can achieve nearly perfect accuracy(within 5%) while requiring human intervention on less than 30% of the queries.

通过引入基于类别不对称的模型级联框架(IDK)，充分利用预训练模型集合，实现深度学习推断加速和成本降低，无需重新训练模型。在多个基准测试中验证了该框架的有效性。

IDK Cascades：通过避免过分思考加速深度学习