Abstract:Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.

Subbatch learning method for BP neural networks

A Learning Algorithm for Solving the Over-Fitting Problem in BPANN

Evolving Neural Networks Using the Hybrid of Ant Colony Optimization and Bp Algorithms

Barzilai-Borwein-based Adaptive Learning Rate for Deep Learning

The Study of Approximate Optimal Step Size of B-P Neural Network

Research and Application on BP Neural Network Algorithm

An Effective Backpropagation Algorithm for Optimizing BP Neural Network Based on Rough Set and Modified Genetic Algorithm

Brief Introduction of Back Propagation (BP) Neural Network Algorithm and Its Improvement

An Improved Mini-Batching Technique: Sample-and-Learn

New and Rapidly Converging BP Neural Network Algorithm

Optimal parameters selection of back propagation algorithm in the feedforward neural network

An optimizing BP neural network algorithm based on genetic algorithm

Parameter Optimization and State Evaluation of Basketball Teaching Based on BPNN

Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks

Optimization Study of BP Neural Network Based on Genetic Algorithm

Prediction of Improved BP Neural Network by Adaboost Algorithm

Evolving Deep Neural Networks via Cooperative Coevolution With Backpropagation

Global Convergence of Online Bp Training with Dynamic Learning Rate

Sub-pixel Mapping Method Based on BP Neural Network

Towards Better Generalization: BP-SVRG in Training Deep Neural Networks

Back Propagation With Randomized Cost Function For Training Neural Networks