Running multiple deep neural networks (DNNs) in parallel has become an
emerging workload in both edge devices, such as mobile phones where multiple
tasks serve a single user for daily activities, and data centers, where various
requests are raised from millions of users, as seen with large language models.
To reduce the costly computational and memory requirements of these workloads,
various efficient sparsification approaches have been introduced, resulting in
widespread sparsity across different types of DNN models. In this context,
there is an emerging need for scheduling sparse multi-DNN workloads, a problem
that is largely unexplored in previous literature. This paper systematically
analyses the use-cases of multiple sparse DNNs and investigates the
opportunities for optimizations. Based on these findings, we propose Dysta, a
novel bi-level dynamic and static scheduler that utilizes both static sparsity
patterns and dynamic sparsity information for the sparse multi-DNN scheduling.
Both static and dynamic components of Dysta are jointly designed at the
software and hardware levels, respectively, to improve and refine the
scheduling approach. To facilitate future progress in the study of this class
of workloads, we construct a public benchmark that contains sparse multi-DNN
workloads across different deployment scenarios, spanning from mobile phones
and AR/VR wearables to data centers. A comprehensive evaluation on the sparse
multi-DNN benchmark demonstrates that our proposed approach outperforms the
state-of-the-art methods with up to 10% decrease in latency constraint
violation rate and nearly 4X reduction in average normalized turnaround time.
Our artifacts and code are publicly available at:
this https URL

本文系统分析了多个稀疏深度神经网络的使用情况并研究了优化的机会，基于这些研究结果，我们提出了 Dysta，这是一种利用静态和动态稀疏信息进行稀疏多深度神经网络调度的新型双层动态和静态调度器。我们构建了一个公共基准，其中包含了跨移动电话、AR/VR 可穿戴设备和数据中心的不同部署情景的稀疏多深度神经网络工作负载。对稀疏多深度神经网络基准的全面评估表明我们提出的方法在满足延迟限制方面的违规率减少了 10% 左右，并且平均归一化周转时间减少了近 4 倍。我们的工件和代码可公开获取。