Abstract:Dropout is perhaps the most popular regularization method for deep learning. Due to the stochastic nature of the Dropout mechanism, the convergence analysis of Dropout learning is challenging and the existing convergence results are mainly of probability nature. In this paper, we investigate the deterministic convergence of the mini-batch gradient learning method with Dropconnect and penalty. By drawing and presenting a set of samples of the mask matrix of Dropconnect regularization into the learning process in a cyclic manner, we establish an upper bound of the norm of the weight vector sequence and prove that the gradient of the cost function, the cost function itself, and the weight vector sequence deterministically converge to zero, a constant, and a fixed point respectively. Considering Dropout is mathematically a specific realization of Dropconnect, the established theoretical results in this paper are also valid for Dropout learning. Illustrative simulations on the MNIST dataset are provided to verify the theoretical analysis.

Boundedness and Convergence of Batch Back-Propagation Algorithm with Penalty for Feedforward Neural Networks

Convergence of batch BP algorithm with penalty for FNN training

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

Relaxed Conditions for Convergence of Batch BPAP for Feedforward Neural Networks.

Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty

Boundedness and Convergence of Online Gradient Method with Penalty and Momentum

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Convergence of Batch Gradient Learning with Smoothing Regularization and Adaptive Momentum for Neural Networks

Convergence of A Gradient Algorithm with Penalty for Training Two-Layer Neural Networks

Strong Convergence Of Gradient Methods For Bp Networks Training

Convergence and monotonicity of an online gradient method with penalty for neural networks

Convergence Analysis of Multilayer Feedforward Networks Trained with Penalty Terms: A review

Convergence of batch gradient algorithm for feedforward neural network training

Computational Properties and Convergence Analysis of BPNN for Cyclic and Almost Cyclic Learning with Penalty

Convergence of Online Gradient Method with a Penalty Term for BP Neural Network with Stochastic Inputs

Boundedness and Convergence of Mini-batch Gradient Method with Cyclic Dropconnect and Penalty

Convergence of Chaos Injection-Based Batch Backpropagation Algorithm for Feedforward Neural Networks

Boundedness and convergence of MPN for cyclic and almost cyclic learning with penalty

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Convergence of BP Algorithm with Variable Learning Rates for FNN Training