Abstract:A good weight initialization is crucial to accelerate the convergence of the weights in a neural network. However, training a neural network is still time-consuming, despite recent advances in weight initialization approaches. In this paper, we propose a mathematical framework for the weight initialization in the last layer of a neural network. We first derive analytically a tight constraint on the weights that accelerates the convergence of the weights during the back-propagation algorithm. We then use linear regression and Lagrange multipliers to analytically derive the optimal initial weights and initial bias of the last layer, that minimize the initial training loss given the derived tight constraint. We also show that the restrictive assumption of traditional weight initialization algorithms that the expected value of the weights is zero is redundant for our approach. We first apply our proposed weight initialization approach to a Convolutional Neural Network that predicts the Remaining Useful Life of aircraft engines. The initial training and validation loss are relatively small, the weights do not get stuck in a local optimum, and the convergence of the weights is accelerated. We compare our approach with several benchmark strategies. Compared to the best performing state-of-the-art initialization strategy (Kaiming initialization), our approach needs 34% less epochs to reach the same validation loss. We also apply our approach to ResNets for the CIFAR-100 dataset, combined with transfer learning. Here, the initial accuracy is already at least 53%. This gives a faster weight convergence and a higher test accuracy than the benchmark strategies.

Using linear initialisation to improve speed of convergence and fully-trained error in Autoencoders

A mathematical framework for improved weight initialization of neural networks using Lagrange multipliers

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

A Sober Look at Neural Network Initializations

A NEW TRAINING ALGORITHM OF AUTOENCODER COMBINING LATERAL INHIBITION MECHANISM

A weight initialization based on the linear product structure for neural networks

Improved weight initialization for deep and narrow feedforward neural network

Initialization Approach for Nonlinear State-Space Identification via the Subspace Encoder Approach

On weight initialization in deep neural networks

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Improving performance of recurrent neural network with relu nonlinearity

Eigenvalue initialisation and regularisation for Koopman autoencoders

Early alignment in two-layer networks training is a two-edged sword

Comparison of Initial Learning Algorithms for Long Short-Term Memory Method on Real-Time Respiratory Signal Prediction

On the Role of Initialization on the Implicit Bias in Deep Linear Networks

Weight initialization based‐rectified linear unit activation function to improve the performance of a convolutional neural network model

Tuning the Weights: The Impact of Initial Matrix Configurations on Successor Features Learning Efficacy

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Weight Conditioning for Smooth Optimization of Neural Networks

Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks

Improved Initialization of Pre-Training in Deep Neural Network