Reducing the latency variance in machine learning inference is a key
requirement in many applications. Variance is harder to control in a cloud
deployment in the presence of stragglers. In spite of this challenge, inference
is increasingly being done in the cloud, due to the advent of affordable
machine learning as a service (MLaaS) platforms. Existing approaches to reduce
variance rely on replication which is expensive and partially negates the
affordability of MLaaS. In this work, we argue that MLaaS platforms also
provide unique opportunities to cut the cost of redundancy. In MLaaS platforms,
multiple inference requests are concurrently received by a load balancer which
can then create a more cost-efficient redundancy coding across a larger
collection of images. We propose a novel convolutional neural network model,
Collage-CNN, to provide a low-cost redundancy framework. A Collage-CNN model
takes a collage formed by combining multiple images and performs multi-image
classification in one shot, albeit at slightly lower accuracy. We then augment
a collection of traditional single image classifiers with a single Collage-CNN
classifier which acts as a low-cost redundant backup. Collage-CNN then provides
backup classification results if a single image classification straggles.
Deploying the Collage-CNN models in the cloud, we demonstrate that the 99th
percentile tail latency of inference can be reduced by 1.47X compared to
replication based approaches while providing high accuracy. Also, variation in
inference latency can be reduced by 9X with a slight increase in average
inference latency.

提出了一种使用 Collage-CNN 模型结合 MLaaS 平台的低成本冗余框架来降低 Inference 的尾延迟，并降低推理延迟的变异性的方法。实验表明，相对于以复制为基础的方法，可以将推理的 99-percentile 尾延迟降低 1.47 倍，同时保持高准确性。