Affiliations: [a] Artificial Intelligence Department, China Academy of Information and Communication Technology (CAICT), Haidian District, Beijing, China
| [b] Research and Development Department, Beijing Qinghong Technology Co., Ltd., Chaoyang District, Beijing, China
Correspondence:
[*]
Corresponding author. Wei Min Zhang, Artificial Intelligence Department, China Academy of Information and Communication Technology (CAICT), No. 52 Huayuan Bei Road, Haidian District, Beijing, China. E-mail: zhangweimin@caict.ac.cn.
Note: [1] IBD is short for Inference Benchmark for DNN processor, this research has been partially supported by the AIIA DNN benchmark project, and open source tools can be find in: https://github.com/AIIABenchmark/AIIA-DNN-benchmark.
Abstract: With the many varieties of AI hardware prevailing on the market, it is often hard to decide which one is the most suitable to use but not only with the best performance. As there is an industry-wide trend demand for deep learning deployment, the inference benchmark for the effectiveness of DNN processor becomes important and is of great help to select and optimize AI hardware. To systematically benchmark deep learning deployment platforms, and give more objective and useful metrics comparison. In this paper, an end to end benchmark evaluation system was brought up called IBD, it combined 4 steps include three components with 6 metrics. The performance comparison results are obtained from the chipsets from Qualcomm, HiSilicon, and NVIDIA, which can provide hardware acceleration for AI inference. To comprehensively reflect the current status of the DNN processor deploying performance, we chose six devices from three kinds of deployment scenarios which are cloud, desktop and mobile, ten models from three different kinds of applications with diverse characteristics are selected, and all these models are trained from three major training frameworks. Several important observations were made by using our methodologies. Experimental results showed that workload diversity should focus on the difference came from training frameworks, inference frameworks with specific processors, input size and precision (floating and quantized).
Keywords: AI, deep neural network processor, benchmark, end to end, inference
