Abstract:Model optimization in deep learning (DL) and neural networks is concerned about how and why the model can be successfully trained towards one or more objective functions. The evolutionary learning or training process continuously considers the dynamic parameters of the model. Many researchers propose a deep learning-based solution by randomly selecting a single classifier model architecture. Such approaches generally overlook the hidden and complex nature of the model’s internal working, producing biased results. Larger and deeper NN models bring many complexities and logistic challenges while building and deploying them. To obtain high-quality performance results, an optimal model generally depends on the appropriate architectural settings, such as the number of hidden layers and the number of neurons at each layer. A challenging and time-consuming task is to select and test various combinations of these settings manually. This paper presents an extensive empirical analysis of various deep learning algorithms trained recursively using permutated settings to establish benchmarks and find an optimal model. The paper analyzed the Stack Overflow dataset to predict the quality of posted questions. The extensive empirical analysis revealed that some famous deep learning algorithms such as CNN are the least effective algorithm in solving this problem compared to multilayer perceptron (MLP), which provides efficient computing and the best results in terms of prediction accuracy. The analysis also shows that manipulating the number of neurons alone at each layer in a network does not influence model optimization. This paper’s findings will help to recognize the fact that future models should be built by considering a vast range of model architectural settings for an optimal solution.

Neural Network Architecture Optimization through Submodularity and Supermodularity.

Automatically Design Convolutional Neural Networks by Optimization With Submodularity and Supermodularity

Optimizing Recurrent Neural Networks Architectures under Time Constraints.

Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?

Neural architecture optimization

Optimization of deep learning models: benchmark and analysis

Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization

Accelerating Neural Architecture Search using Performance Prediction

NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling

Optimizing Deep Feedforward Neural Network Architecture: A Tabu Search Based Approach

Learning Interpretable Models Through Multi-Objective Neural Architecture Search

Optimal Deep Neural Networks by Maximization of the Approximation Power

Mixed Precision Neural Architecture Search for Energy Efficient Deep Learning.

ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation

Multi-Objective Neural Architecture Search for Light-Weight Model

Co-Evolutionary Modular Neural Networks for Automatic Problem Decomposition

Training compact neural networks via

An efficient optimization approach for designing machine learning models based on genetic algorithm

Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach

Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training