There is a growing demand to deploy computation-intensive deep learning (DL) models on resource-constrained mobile devices for real-time intelligent applications. Equipped with a variety of processing units such as CPUs, GPUs, and NPUs, the mobile devices hold potential to accelerate DL inference via parallel execution across heterogeneous processors. Various efficient parallel methods have been explored to optimize computation distribution, achieve load balance, and minimize communication cost across processors. Yet their practical effectiveness in the dynamic and diverse real-world mobile environment is less explored. This paper presents a holistic empirical study to assess the capabilities and challenges associated with parallel DL inference on heterogeneous mobile processors. Through carefully designed experiments covering various DL models, mobile software/hardware environments, workload patterns, and resource availability, we identify limitations of existing techniques and highlight opportunities for cross-level optimization.

在资源受限的移动设备上部署计算密集型的深度学习模型以实现实时智能应用的需求与日俱增，在各种处理单元（如CPU、GPU和NPU）的支持下，移动设备有潜力通过在异构处理器之间进行并行执行来加速深度学习推理。本文通过精心设计的实验，涵盖了各种深度学习模型、移动软件/硬件环境、工作负载模式和资源可用性，综合性地评估了在异构移动处理器上进行并行深度学习推理的能力和挑战，并确定了现有技术的局限性，并强调了跨层级优化的机会。

异构移动处理器上的深度学习推理：潜力与陷阱