Abstract:Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Impact of Parameter Tuning for Optimizing Deep Neural Network Models for Predicting Software Faults

A systematic review of hyperparameter tuning techniques for software quality prediction models

Hyperparameter Tuning of Deep learning Models in Keras

Comparative Study of Various Hyperparameter Tuning on Random Forest Classification With SMOTE and Feature Selection Using Genetic Algorithm in Software Defect Prediction

Estimation of Small Failure Probability Based on Adaptive Subset Simulation and Deep Neural Network

Performance Comparison of Various Algorithms During Software Fault Prediction

An effective algorithm for hyperparameter optimization of neural networks

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks

Optimal Kernel Tuning Parameter Prediction using Deep Sequence Models

Augmentation of Decision Tree Model Through Hyper-Parameters Tuning for Monitoring of Cutting Tool Faults Based on Vibration Signatures

Hybrid Optimization-Based Neural Network Classifier for Software Defect Prediction

Using a One-Dimensional Convolutional Neural Network with Taguchi Parametric Optimization for a Permanent-Magnet Synchronous Motor Fault-Diagnosis System

Machine Learning to Aid Tuning of Numerical Parameters in Topology Optimization

Auto-Tuning Spark Configurations Based On Neural Network

A Survey on Multi-Objective based Parameter Optimization for Deep Learning

Predicting the Number of Software Faults using Deep Learning

Is One Hyperparameter Optimizer Enough?

A New Improved Prediction of Software Defects Using Machine Learning-based Boosting Techniques with NASA Dataset

Optimizing a Digital Twin for Fault Diagnosis in Grid Connected Inverters -- A Bayesian Approach

Hyper-parameter optimization of deep learning model for prediction of Parkinson’s disease

The Effect of Hyperparameter Optimization on the Estimation of Performance Metrics in Network Traffic Prediction using the Gradient Boosting Machine Model