Fei Tang,Wanling Gao,Jianfeng Zhan,Chuanxin Lan,Xu Wen,Lei Wang,Chunjie Luo,Jiahui Dai,Zheng Cao,Xingwang Xiong,Zihan Jiang,Tianshu Hao,Fanda Fan,Fan Zhang,Yunyou Huang,Jianan Chen,Mengjia Du,Rui Ren,Chen Zheng,Daoyi Zheng,Haoning Tang,Kunlin Zhan,Biao Wang,Defei Kong,Minghe Yu,Chongkang Tan,Huan Li,Xinhui Tian,Yatao Li,Junchao Shao,Zhenyu Wang,Xiaoyu Wang,Hainan Ye

Abstract:The booming successes of machine learning in different domains boost industry-scale deployments of innovative AI algorithms, systems, and architectures, and thus the importance of benchmarking grows. However, the confidential nature of the workloads, the paramount importance of the representativeness and diversity of benchmarks, and the prohibitive cost of training a state-of-the-art model mutually aggravate the AI benchmarking challenges. In this paper, we present a balanced AI benchmarking methodology for meeting the subtly different requirements of different stages in developing a new system/architecture and ranking/purchasing commercial off-the-shelf ones. Performing an exhaustive survey on the most important AI domain-Internet services with seventeen industry partners, we identify and include seventeen representative AI tasks to guarantee the representativeness and diversity of the benchmarks. Meanwhile, for reducing the benchmarking cost, we select a benchmark subset to a minimum-three tasks-according to the criteria: diversity of model complexity, computational cost, and convergence rate, repeatability, and having widely-accepted metrics or not. We contribute by far the most comprehensive AI benchmark suite-AIBench. The evaluations show AIBench outperforms MLPerf in terms of the diversity and representativeness of model complexity, computational cost, convergent rate, computation and memory access patterns, and hotspot functions. With respect to the AIBench full benchmarks, its subset shortens the benchmarking cost by 41%, while maintaining the primary workload characteristics. The specifications, source code, and performance numbers are publicly available from the web site this http URL.

Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics

AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite

Aibench: an industry standard ai benchmark suite

AIBench: an Industry Standard AI Benchmark Suite from Internet Services.

Benchmarking State-of-the-Art Deep Learning Software Tools

P F ] 1 3 A ug 2 01 9 HPC AI 500 : A Benchmark Suite for HPC AI Systems

Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking.

Various Frameworks and Libraries of Machine Learning and Deep Learning: A Survey

Scientific Machine Learning Benchmarks

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning

Bosch Deep Learning Hardware Benchmark

Benchmarking of DL Libraries and Models on Mobile Devices

A Comprehensive Benchmark of Deep Learning Libraries on Mobile Devices

Quantitative Evaluation of Deep Learning Frameworks in Heterogeneous Computing Environment

DataPerf: Benchmarks for Data-Centric AI Development

A Survey on Hardware Accelerator Design of Deep Learning for Edge Devices

DLBench: An Experimental Evaluation of Deep Learning Frameworks

BENCHIP： Benchmarking Intelligence Processors

Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training