Currently, there is a growing trend of outsourcing the execution of DNNs to
cloud services. For service providers, managing multi-tenancy and ensuring
high-quality service delivery, particularly in meeting stringent execution time
constraints, assumes paramount importance, all while endeavoring to maintain
cost-effectiveness. In this context, the utilization of heterogeneous
multi-accelerator systems becomes increasingly relevant. This paper presents
RELMAS, a low-overhead deep reinforcement learning algorithm designed for the
online scheduling of DNNs in multi-tenant environments, taking into account the
dataflow heterogeneity of accelerators and memory bandwidths contentions. By
doing so, service providers can employ the most efficient scheduling policy for
user requests, optimizing Service-Level-Agreement (SLA) satisfaction rates and
enhancing hardware utilization. The application of RELMAS to a heterogeneous
multi-accelerator system composed of various instances of Simba and Eyeriss
sub-accelerators resulted in up to a 173% improvement in SLA satisfaction rate
compared to state-of-the-art scheduling techniques across different workload
scenarios, with less than a 1.5% energy overhead.

当前，云服务外包 DNNs 的趋势日益增长。本文介绍了一种针对多租户环境中 DNNs 在线调度的低开销深度强化学习算法 RELMAS，考虑了加速器的数据流异构性和内存带宽竞争问题，提高了服务提供商的硬件利用率并优化了服务水平协议满意率。在包含 Simba 和 Eyeriss 子加速器实例的多异构加速器系统上，与最先进的调度技术相比，在不同工作负载场景下，SLA 满意率提高了 173％，能源开销仅增加了不到 1.5％。