Image recognition tasks typically use deep learning and require enormous
processing power, thus relying on hardware accelerators like GPUs and TPUs for
fast, timely processing. Failure in real-time image recognition tasks can occur
due to sub-optimal mapping on hardware accelerators during model deployment,
which may lead to timing uncertainty and erroneous behavior. Mapping on
hardware accelerators is done through multiple software components like deep
learning frameworks, compilers, device libraries, that we refer to as the
computational environment. Owing to the increased use of image recognition
tasks in safety-critical applications like autonomous driving and medical
imaging, it is imperative to assess their robustness to changes in the
computational environment, as the impact of parameters like deep learning
frameworks, compiler optimizations, and hardware devices on model performance
and correctness is not well understood.
In this paper we present a differential testing framework, which allows deep
learning model variant generation, execution, differential analysis and testing
for a number of computational environment parameters. Using our framework, we
conduct an empirical study of robustness analysis of three popular image
recognition models using the ImageNet dataset, assessing the impact of changing
deep learning frameworks, compiler optimizations, and hardware devices. We
report the impact in terms of misclassifications and inference time differences
across different settings. In total, we observed up to 72% output label
differences across deep learning frameworks, and up to 82% unexpected
performance degradation in terms of inference time, when applying compiler
optimizations. Using the analysis tools in our framework, we also perform fault
analysis to understand the reasons for the observed differences.

本文提出了一种差分测试框架，该框架可对多种计算环境参数进行深度学习模型变量生成、执行、差分分析和测试，对三种热门图像识别模型使用 ImageNet 数据集进行鲁棒性分析，评估了更改深度学习框架、编译器优化和硬件设备的影响。