Abstract:Recent advances in deep neural networks (DNNs) owe their success to training algorithms that use backpropagation and gradient-descent. Backpropagation, while highly effective on von Neumann architectures, becomes inefficient when scaling to large networks. Commonly referred to as the weight transport problem, each neuron's dependence on the weights and errors located deeper in the network require exhaustive data movement which presents a key problem in enhancing the performance and energy-efficiency of machine-learning hardware. In this work, we propose a bio-plausible alternative to backpropagation drawing from advances in feedback alignment algorithms in which the error computation at a single synapse reduces to the product of three scalar values. Using a sparse feedback matrix, we show that a neuron needs only a fraction of the information previously used by the feedback alignment algorithms. Consequently, memory and compute can be partitioned and distributed whichever way produces the most efficient forward pass so long as a single error can be delivered to each neuron. Our results show orders of magnitude improvement in data movement and $2\times$ improvement in multiply-and-accumulate operations over backpropagation. Like previous work, we observe that any variant of feedback alignment suffers significant losses in classification accuracy on deep convolutional neural networks. By transferring trained convolutional layers and training the fully connected layers using direct feedback alignment, we demonstrate that direct feedback alignment can obtain results competitive with backpropagation. Furthermore, we observe that using an extremely sparse feedback matrix, rather than a dense one, results in a small accuracy drop while yielding hardware advantages. All the code and results are available under <a class="link-external link-https" href="https://github.com/bcrafton/ssdfa" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the weight transport problem existing in the training process of deep neural networks (DNNs). Specifically, although the traditional Backpropagation (BP) algorithm is very effective on the von Neumann architecture, it becomes inefficient when extended to large - scale networks. This is because each neuron depends on the weights and errors in the deeper layers of the network, which requires a large amount of data movement, thus affecting the performance and energy efficiency of machine - learning hardware. To solve this problem, the paper proposes a Sparse Direct Feedback Alignment (SDFA) method based on sparse connections. This method reduces the amount of computation, memory usage and data movement required for neurons to update weights while maintaining high classification accuracy. The main contributions of the paper are as follows: 1. **Proposing a biologically - plausible alternative**: By drawing inspiration from the Feedback Alignment (FA) algorithm, the paper proposes a novel method, that is, using a sparse feedback matrix to reduce the error calculation on a single synapse, which is simplified to the product of three scalar values. 2. **Reducing data movement and computational complexity**: Compared with traditional backpropagation, SDFA significantly reduces the amount of data movement and the number of multiply - accumulate operations, especially when dealing with large - scale networks. 3. **Maintaining high classification accuracy**: Although a sparse feedback matrix is used, the paper experimentally proves that this sparsity has a negligible impact on network performance, especially when the feedback matrix is close to full - rank or completely full - rank. 4. **Applicable to multiple standard data sets**: The paper evaluates the proposed algorithm on multiple standard data sets (such as ImageNet), proving its effectiveness in dealing with complex problems. Through these improvements, the paper aims to promote the development of neural network training algorithms, making them more efficient, energy - saving and more suitable for the needs of future intelligent systems.

Direct Feedback Alignment with Sparse Connections for Local Learning

Direct Feedback Alignment With Sparse Connections for Local Learning

Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

Forward Direct Feedback Alignment for Online Gradient Estimates of Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Training Spiking Neural Networks via Augmented Direct Feedback Alignment

IM-LIF: Improved Neuronal Dynamics with Attention Mechanism for Direct Training Deep Spiking Neural Network

LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization

Deep Reservoir Networks with Learned Hidden Reservoir Weights using Direct Feedback Alignment

Accelerated Local Training of CNNs by Optimized Direct Feedback Alignment Based on Stochasticity of 4 Mb C-doped Ge<sub>2</sub>Sb<sub>2</sub>Te<sub>5</sub> PCM Chip in 40 nm Node

Learning with augmented target information: An alternative theory of Feedback Alignment

Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks

Scaling Laws Beyond Backpropagation

Interlocking Backpropagation: Improving depthwise model-parallelism

Learning efficient backprojections across cortical hierarchies in real time

Optical training of large-scale Transformers and deep neural networks with direct feedback alignment

Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

Efficient Training Convolutional Neural Networks on Edge Devices with Gradient-pruned Sign-symmetric Feedback Alignment

Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?

Optical Training Framework for Optical Diffractive Deep Neural Network Via Direct Feedback Alignment