Recently, there has been an explosive growth of mobile and embedded
applications using convolutional neural networks(CNNs). To alleviate their
excessive computational demands, developers have traditionally resorted to
cloud offloading, inducing high infrastructure costs and a strong dependence on
networking conditions. On the other end, the emergence of powerful SoCs is
gradually enabling on-device execution. Nonetheless, low- and mid-tier
platforms still struggle to run state-of-the-art CNNs sufficiently. In this
paper, we present DynO, a distributed inference framework that combines the
best of both worlds to address several challenges, such as device
heterogeneity, varying bandwidth and multi-objective requirements. Key
components that enable this are its novel CNN-specific data packing method,
which exploits the variability of precision needs in different parts of the CNN
when onloading computation, and its novel scheduler that jointly tunes the
partition point and transferred data precision at run time to adapt inference
to its execution environment. Quantitative evaluation shows that DynO
outperforms the current state-of-the-art, improving throughput by over an order
of magnitude over device-only execution and up to 7.9x over competing CNN
offloading systems, with up to 60x less data transferred.

本文介绍了一个名为 DynO 的分布式推理框架，它结合了云端卸载计算和设备端计算的最佳优势，采用新颖的 CNN 数据打包方法和调度器，在运行时联合调整分区点和传输数据精度来适应执行环境，并且评估结果表明，DynO 优于当前最先进的系统，在设备端执行时提高吞吐量超过一个数量级，在竞争的 CNN 卸载系统中提高了 7.9 倍，并且传输的数据量减少了高达 60 倍。