Deep Learning (DL) has developed to become a corner-stone in many everyday
applications that we are now relying on. However, making sure that the DL model
uses the underlying hardware efficiently takes a lot of effort. Knowledge about
inference characteristics can help to find the right match so that enough
resources are given to the model, but not too much. We have developed a DL
Inference Performance Predictive Model (DIPPM) that predicts the inference
latency, energy, and memory usage of a given input DL model on the NVIDIA A100
GPU. We also devised an algorithm to suggest the appropriate A100
Multi-Instance GPU profile from the output of DIPPM. We developed a methodology
to convert DL models expressed in multiple frameworks to a generalized graph
structure that is used in DIPPM. It means DIPPM can parse input DL models from
various frameworks. Our DIPPM can be used not only helps to find suitable
hardware configurations but also helps to perform rapid design-space
exploration for the inference performance of a model. We constructed a graph
multi-regression dataset consisting of 10,508 different DL models to train and
evaluate the performance of DIPPM, and reached a resulting Mean Absolute
Percentage Error (MAPE) as low as 1.9%.

本文介绍了一种基于深度学习推理性能预测模型的方法，该模型可以预测在 NVIDIA A100 GPU 上给定输入的深度学习模型的推理延迟、能量和内存使用情况，并建议适当的 GPU 配置方案，同时帮助进行模型推理性能的快速设计空间探索。