Abstract:Backpropagation (BP) has been a successful optimization technique for deep learning models. However, its limitations, such as backward- and update-locking, and its biological implausibility, hinder the concurrent updating of layers and do not mimic the local learning processes observed in the human brain. To address these issues, recent research has suggested using local error signals to asynchronously train network blocks. However, this approach often involves extensive trial-and-error iterations to determine the best configuration for local training. This includes decisions on how to decouple network blocks and which auxiliary networks to use for each block. In our work, we introduce a novel BP-free approach: a block-wise BP-free (BWBPF) neural network that leverages local error signals to optimize distinct sub-neural networks separately, where the global loss is only responsible for updating the output layer. The local error signals used in the BP-free model can be computed in parallel, enabling a potential speed-up in the weight update process through parallel implementation. Our experimental results consistently show that this approach can identify transferable decoupled architectures for VGG and ResNet variations, outperforming models trained with end-to-end backpropagation and other state-of-the-art block-wise learning techniques on datasets such as CIFAR-10 and Tiny-ImageNet. The code is released at <a class="link-external link-https" href="https://github.com/Belis0811/BWBPF" rel="external noopener nofollow">this https URL</a>.

Bp Learning Algorithm Based on Dfp and Trust Region Method

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A Learning Algorithm for Solving the Over-Fitting Problem in BPANN

Evolving Neural Networks Using the Hybrid of Ant Colony Optimization and Bp Algorithms

Research and Application on BP Neural Network Algorithm

Brief Introduction of Back Propagation (BP) Neural Network Algorithm and Its Improvement

New and Rapidly Converging BP Neural Network Algorithm

Research of New Learning Method of Feedforward Neural Network

Optimal parameters selection of back propagation algorithm in the feedforward neural network

The Improved Rapid Convergence Algorithm of the Connecting Rights in the BP Network

Back Propagation With Randomized Cost Function For Training Neural Networks

Fast Learning Algorithms for Feedforward Neural Networks

A Modified Learning Algorithm for Improving the Fault Tolerance of BP Networks

An optimizing BP neural network algorithm based on genetic algorithm

Dynamical Optimal Learning For Fnn And Its Applications

A Fast Learning Algorithm of Neural Networks by Changing Error Functions

A Fast Learning Algorithm Of Neural Network For The Training And Recognition Of The Phonemes

Understanding Predictive Coding as an Adaptive Trust-Region Method

Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks

Convergence Analysis of Novel Fractional-Order Backpropagation Neural Networks With Regularization Terms

Using Fitness Dependent Optimizer for Training Multi-layer Perceptron