Abstract:Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Optimize the Performance of the Neural Network by using a Mini Dataset Processing Method

Survey on Energy-Efficient Deep Neural Networks for Computer Vision

Can we learn better with hard samples?

A methodological framework for optimizing the energy consumption of deep neural networks: a case study of a cyber threat detector

An Efficient Optimization Technique for Training Deep Neural Networks

A Model Compression Method Using Significant Data and Knowledge Distillation

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks

A Practical Approach to the Analysis and Optimization of Neural Networks on Embedded Systems

Optimizing Dense Feed-Forward Neural Networks

Training Artificial Neural Networks Using a Global Optimization Method That Utilizes Neural Networks

Reducing Data Bottlenecks in Distributed, Heterogeneous Neural Networks

Optimization of Convolutional Neural Network using Microcanonical Annealing Algorithm

Data-Efficient Augmentation for Training Neural Networks

Efficient Neural Network Training via Subset Pretraining

Optimization of deep learning models: benchmark and analysis

Optimizing for In-memory Deep Learning with Emerging Memory Technology

Metaheuristic Algorithms for Convolution Neural Network

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

A Data-Centric Approach to improve performance of deep learning models

Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation

High-speed hyperparameter optimization for deep ResNet models in image recognition