Currently, there is a growing trend of outsourcing the execution of DNNs to
cloud services. For service providers, managing multi-tenancy and ensuring
high-quality service delivery, particularly in meeting stringent execution time
constraints, assumes paramount importance, all while endeavoring to maintain
cost-effectiveness. In this context, the utilization of heterogeneous
multi-accelerator systems becomes increasingly relevant. This paper presents
RELMAS, a low-overhead deep reinforcement learning algorithm designed for the
online scheduling of DNNs in multi-tenant environments, taking into account the
dataflow heterogeneity of accelerators and memory bandwidths contentions. By
doing so, service providers can employ the most efficient scheduling policy for
user requests, optimizing Service-Level-Agreement (SLA) satisfaction rates and
enhancing hardware utilization. The application of RELMAS to a heterogeneous
multi-accelerator system composed of various instances of Simba and Eyeriss
sub-accelerators resulted in up to a 173% improvement in SLA satisfaction rate
compared to state-of-the-art scheduling techniques across different workload
scenarios, with less than a 1.5% energy overhead.

当前，云服务外包 DNNs 的趋势日益增长。本文介绍了一种针对多租户环境中 DNNs 在线调度的低开销深度强化学习算法 RELMAS，考虑了加速器的数据流异构性和内存带宽竞争问题，提高了服务提供商的硬件利用率并优化了服务水平协议满意率。在包含 Simba 和 Eyeriss 子加速器实例的多异构加速器系统上，与最先进的调度技术相比，在不同工作负载场景下，SLA 满意率提高了 173％，能源开销仅增加了不到 1.5％。

基于深度强化学习的深度神经网络多租户多加速器系统的在线调度策略

Deep Reinforcement Learning based Online Scheduling Policy for Deep  Neural Network Multi-Tenant Multi-Accelerator Systems

This paper addresses the critical challenge of managing Quality of Service
(QoS) in cloud services, focusing on the nuances of individual tenant
expectations and varying Service Level Indicators (SLIs). It introduces a novel
approach utilizing Deep Reinforcement Learning for tenant-specific QoS
management in multi-tenant, multi-accelerator cloud environments. The chosen
SLI, deadline hit rate, allows clients to tailor QoS for each service request.
A novel online scheduling algorithm for Deep Neural Networks in
multi-accelerator systems is proposed, with a focus on guaranteeing
tenant-wise, model-specific QoS levels while considering real-time constraints.

这篇论文解决了云服务中管理服务质量（QoS）的关键挑战，着重研究了个别租户期望和不同服务水平指标（SLI）的细微差别。它引入了一种新的方法，利用深度强化学习在多租户、多加速器的云环境中进行租户特定的 QoS 管理。所选择的 SLI，即截止时间满足率，允许客户为每个服务请求量身定制服务质量。提出了一种新颖的在线调度算法，针对多加速器系统中的深度神经网络，着重保证租户层面和模型特定的 QoS 水平，并考虑实时约束。