Two distinguishing features of state-of-the-art mobile and autonomous systems are 1) there are often multiple workloads, mainly deep neural network (DNN) inference, running concurrently and continuously; and 2) they operate on shared memory system-on-chips (SoC) that embed heterogeneous accelerators tailored for specific operations. State-of-the-art lacks efficient performance and resource management techniques necessary to either maximize total system throughput or minimize end-to-end workload latency. In this work, we propose HaX-CoNN, a novel scheme that characterizes and maps layers in concurrently executing DNN inference workloads to a diverse set of accelerators within a SoC. Our scheme uniquely takes per-layer execution characteristics, shared memory (SM) contention, and inter-accelerator transitions into account to find optimal schedules. We evaluate HaX-CoNN on NVIDIA Orin, NVIDIA Xavier, and Qualcomm Snapdragon 865 SoCs. Our experimental results indicate that HaX-CoNN minimizes memory contention by up to 45% and can improve latency and total throughput by up to 32% and 29%, respectively, compared to the state-of-the-art approaches.

本研究提出了一种名为HaX-CoNN的新方案，将并发执行的深度神经网络推理工作负载的层映射到系统级芯片内的多种异构加速器，以最小化内存争用并提升工作负载的延迟和总体吞吐量。实验结果表明，与最先进的方法相比，HaX-CoNN可将内存争用降低最多45%，延迟和总体吞吐量分别提高最多32%和29%。

共享内存冲突感知的异构片上系统并发DNN执行