Abstract:Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

SmartTuning: Selecting Hyper-Parameters of a ConvNet System for Fast Training and Small Working Memory

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning

Training compact neural networks via

FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning with Partitioning and Parallelism of Search Space

Expanding Sparse Tuning for Low Memory Usage

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Growing a Brain: Fine-Tuning by Increasing Model Capacity

Training Compact Neural Networks via Auxiliary Overparameterization

Memory-Tuning: A Unified Parameter-Efficient Tuning Method for Pre-trained Language Models

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks

Discovering Long-Term Effects on Parameter Efficient Fine-tuning

Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference

Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks

Adapting Shortcut with Normalizing Flow: An Efficient Tuning Framework for Visual Recognition

Low-Memory Neural Network Training: A Technical Report

SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models

Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models

Prototypical Fine-Tuning: Towards Robust Performance under Varying Data Sizes

SmartDeal: Remodeling Deep Network Weights for Efficient Inference and Training

Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training