Machine learning (ML) models are increasingly deployed to production, calling
for efficient inference serving systems. Efficient inference serving is
complicated by two challenges: (i) ML models incur high computational costs,
and (ii) the request arrival rates of practical applications have frequent,
high, and sudden variations which make it hard to correctly provision hardware.
Model cascades are positioned to tackle both of these challenges, as they (i)
save work while maintaining accuracy, and (ii) expose a high-resolution
trade-off between work and accuracy, allowing for fine-grained adjustments to
request arrival rates. Despite their potential, model cascades haven't been
used inside an online serving system. This comes with its own set of
challenges, including workload adaption, model replication onto hardware,
inference scheduling, request batching, and more. In this work, we propose
CascadeServe, which automates and optimizes end-to-end inference serving with
cascades. CascadeServe operates in an offline and online phase. In the offline
phase, the system pre-computes a gear plan that specifies how to serve
inferences online. In the online phase, the gear plan allows the system to
serve inferences while making near-optimal adaptations to the query load at
negligible decision overheads. We find that CascadeServe saves 2-3x in cost
across a wide spectrum of the latency-accuracy space when compared to
state-of-the-art baselines on different workloads.

CascadeServe 通过使用模型级联进行端到端的推理服务自动化和优化，在不同工作负载上与现有技术进行比较时，在延迟 - 准确性空间的广泛范围内可以节约 2-3 倍的成本。

CascadeServe: 解锁模型级联进行推理服务

CascadeServe: Unlocking Model Cascades for Inference Serving

Advances in deep learning have led to substantial increases in prediction
accuracy but have been accompanied by increases in the cost of rendering
predictions. We conjecture that fora majority of real-world inputs, the recent
advances in deep learning have created models that effectively "overthink" on
simple inputs. In this paper, we revisit the classic question of building model
cascades that primarily leverage class asymmetry to reduce cost. We introduce
the "I Don't Know"(IDK) prediction cascades framework, a general framework to
systematically compose a set of pre-trained models to accelerate inference
without a loss in prediction accuracy. We propose two search based methods for
constructing cascades as well as a new cost-aware objective within this
framework. The proposed IDK cascade framework can be easily adopted in the
existing model serving systems without additional model re-training. We
evaluate the proposed techniques on a range of benchmarks to demonstrate the
effectiveness of the proposed framework.

通过引入基于类别不对称的模型级联框架 (IDK)，充分利用预训练模型集合，实现深度学习推断加速和成本降低，无需重新训练模型。在多个基准测试中验证了该框架的有效性。