Driven by the wide adoption of deep neural networks (DNNs) across different
application domains, multi-tenancy execution, where multiple DNNs are deployed
simultaneously on the same hardware, has been proposed to satisfy the latency
requirements of different applications while improving the overall system
utilization. However, multi-tenancy execution could lead to undesired
system-level resource contention, causing quality-of-service (QoS) degradation
for latency-critical applications. To address this challenge, we propose MoCA,
an adaptive multi-tenancy system for DNN accelerators. Unlike existing
solutions that focus on compute resource partition, MoCA dynamically manages
shared memory resources of co-located applications to meet their QoS targets.
Specifically, MoCA leverages the regularities in both DNN operators and
accelerators to dynamically modulate memory access rates based on their latency
targets and user-defined priorities so that co-located applications get the
resources they demand without significantly starving their co-runners. We
demonstrate that MoCA improves the satisfaction rate of the service level
agreement (SLA) up to 3.9x (1.8x average), system throughput by 2.3x (1.7x
average), and fairness by 1.3x (1.2x average), compared to prior work.

本文提出了一个基于 MoCA 的自适应多租户系统，该系统通过动态管理共享内存资源来实现不同深度神经网络的优化部署，以提高系统利用率并减少质量服务的下降，通过实验证明 MoCA 可以显著提高服务等级协议的满意度和系统效率，同时减少负载抢占的情况。