Cascade systems comprise a two-model sequence, with a lightweight model
processing all samples and a heavier, higher-accuracy model conditionally
refining harder samples to improve accuracy. By placing the light model on the
device side and the heavy model on a server, model cascades constitute a widely
used distributed inference approach. With the rapid expansion of intelligent
indoor environments, such as smart homes, the new setting of Multi-Device
Cascade is emerging where multiple and diverse devices are to simultaneously
use a shared heavy model on the same server, typically located within or close
to the consumer environment. This work presents MultiTASC, a
multi-tenancy-aware scheduler that adaptively controls the forwarding decision
functions of the devices in order to maximize the system throughput, while
sustaining high accuracy and low latency. By explicitly considering device
heterogeneity, our scheduler improves the latency service-level objective (SLO)
satisfaction rate by 20-25 percentage points (pp) over state-of-the-art cascade
methods in highly heterogeneous setups, while serving over 40 devices,
showcasing its scalability.

本研究提出了 MultiTASC，这是一种多租户感知的调度程序，可以自适应地控制多设备中转发决策功能，以最大化系统吞吐量，同时维持高精度和低延迟。