Deep neural networks (DNNs) have been widely used in various video analytic
tasks. These tasks demand real-time responses. Due to the limited processing
power on mobile devices, a common way to support such real-time analytics is to
offload the processing to an edge server. This paper examines how to speed up
the edge server DNN processing for multiple clients. In particular, we observe
batching multiple DNN requests significantly speeds up the processing time.
Based on this observation, we first design a novel scheduling algorithm to
exploit the batching benefits of all requests that run the same DNN. This is
compelling since there are only a handful of DNNs and many requests tend to use
the same DNN. Our algorithms are general and can support different objectives,
such as minimizing the completion time or maximizing the on-time ratio. We then
extend our algorithm to handle requests that use different DNNs with or without
shared layers. Finally, we develop a collaborative approach to further improve
performance by adaptively processing some of the requests or portions of the
requests locally at the clients. This is especially useful when the network
and/or server is congested. Our implementation shows the effectiveness of our
approach under different request distributions (e.g., Poisson, Pareto, and
Constant inter-arrivals).

本文研究了如何加速深度神经网络在边缘服务器上处理多个客户端的任务，通过批处理的方式和优化调度算法，使用不同的目标函数把多个请求合并，最终提高了处理效率和性能。