Deep learning has become a useful data analysis method, however mainstream
adaption in distributed computer software and embedded devices has been low so
far. Often, adding deep learning inference in mainstream applications and
devices requires new hardware with signal processors suited for convolutional
neural networks. This work adds new data types (quantized 16-bit and 8-bit
integer, 16-bit floating point) to Caffe in order to save memory and increase
inference speed on existing commodity graphics processors with OpenCL, common
in everyday devices. Existing models can be executed effortlessly in
mixed-precision mode. Additionally, we propose a variation of
mixture-of-experts to increase inference speed on AlexNet for image
classification. We managed to decrease memory usage up to 3.29x while
increasing inference speed up to 3.01x on certain devices. We demonstrate with
five simple examples how the presented techniques can easily be applied to
different machine learning problems. The whole pipeline, consisting of models,
example python scripts and modified Caffe library, is available as Open Source
software.

通过添加新的数据类型到 Caffe，增加了现有商品电子设备上深度学习推断速度，在某些设备上，内存使用量可以降低到 3.29 倍，推断速度可以提高到 3.01 倍，并提出了一种混合专家模型的变化来增加图像分类中的推断速度。