Machine learning inference pipelines commonly encountered in data science and
industries often require real-time responsiveness due to their user-facing
nature. However, meeting this requirement becomes particularly challenging when
certain input features require aggregating a large volume of data online.
Recent literature on interpretable machine learning reveals that most machine
learning models exhibit a notable degree of resilience to variations in input.
This suggests that machine learning models can effectively accommodate
approximate input features with minimal discernible impact on accuracy. In this
paper, we introduce Biathlon, a novel ML serving system that leverages the
inherent resilience of models and determines the optimal degree of
approximation for each aggregation feature. This approach enables maximum
speedup while ensuring a guaranteed bound on accuracy loss. We evaluate
Biathlon on real pipelines from both industry applications and data science
competitions, demonstrating its ability to meet real-time latency requirements
by achieving 5.3x to 16.6x speedup with almost no accuracy loss.

介绍了一种名为 Biathlon 的新型 ML 服务系统，利用模型的韧性并确定每个聚合特征的最佳近似度，实现了最大速度提升，同时确保准确性损失的保证边界。通过在工业应用和数据科学竞赛中的实际管道上评估 Biathlon，证明了其在满足实时延迟要求方面的能力，实现了 5.3 倍至 16.6 倍的速度提升，几乎没有准确性损失。