Running multiple deep neural networks (DNNs) in parallel has become an emerging workload in both edge devices, such as mobile phones where multiple tasks serve a single user for daily activities, and data centers, where various requests are raised from millions of users, as seen with large language models. To reduce the costly computational and memory requirements of these workloads, various efficient sparsification approaches have been introduced, resulting in widespread sparsity across different types of DNN models. In this context, there is an emerging need for scheduling sparse multi-DNN workloads, a problem that is largely unexplored in previous literature. This paper systematically analyses the use-cases of multiple sparse DNNs and investigates the opportunities for optimizations. Based on these findings, we propose Dysta, a novel bi-level dynamic and static scheduler that utilizes both static sparsity patterns and dynamic sparsity information for the sparse multi-DNN scheduling. Both static and dynamic components of Dysta are jointly designed at the software and hardware levels, respectively, to improve and refine the scheduling approach. To facilitate future progress in the study of this class of workloads, we construct a public benchmark that contains sparse multi-DNN workloads across different deployment scenarios, spanning from mobile phones and AR/VR wearables to data centers. A comprehensive evaluation on the sparse multi-DNN benchmark demonstrates that our proposed approach outperforms the state-of-the-art methods with up to 10% decrease in latency constraint violation rate and nearly 4X reduction in average normalized turnaround time. Our artifacts and code are publicly available at: https://github.com/SamsungLabs/Sparse-Multi-DNN-Scheduling.

本文系统分析了多个稀疏深度神经网络的使用情况并研究了优化的机会，基于这些研究结果，我们提出了Dysta，这是一种利用静态和动态稀疏信息进行稀疏多深度神经网络调度的新型双层动态和静态调度器。我们构建了一个公共基准，其中包含了跨移动电话、AR/VR可穿戴设备和数据中心的不同部署情景的稀疏多深度神经网络工作负载。对稀疏多深度神经网络基准的全面评估表明我们提出的方法在满足延迟限制方面的违规率减少了10%左右，并且平均归一化周转时间减少了近4倍。我们的工件和代码可公开获取。

Sparse-DySta: 稀疏多DNN工作负载的稀疏感知动态和静态调度