Abstract:One may argue that the simplest type of neural networks beyond a single perceptron is an array of several perceptrons in parallel. In spite of their simplicity, such circuits can compute any Boolean function if one views the majority of the binary perceptron outputs as the binary output of the parallel perceptron, and they are universal approximators for arbitrary continuous functions with values in [0,1] if one views the fraction of perceptrons that output 1 as the analog output of the parallel perceptron. Note that in contrast to the familiar model of a "multi-layer perceptron" the parallel perceptron that we consider here has just binary values as outputs of gates on the hidden layer. For a long time one has thought that there exists no competitive learning algorithm for these extremely simple neural networks, which also came to be known as committee machines. It is commonly assumed that one has to replace the hard threshold gates on the hidden layer by sigmoidal gates (or RBF-gates) and that one has to tune the weights on at least two successive layers in order to achieve satisfactory learning results for any class of neural networks that yield universal approximators. We show that this assumption is not true, by exhibiting a simple learning algorithm for parallel perceptrons - the parallel delta rule (p-delta rule). In contrast to backprop for multi-layer perceptrons, the p-delta rule only has to tune a single layer of weights, and it does not require the computation and communication of analog values with high precision. Reduced communication also distinguishes our new learning rule from other learning rules for parallel perceptrons such as MADALINE. Obviously these features make the p-delta rule attractive as a biologically more realistic alternative to backprop in biological neural circuits, but also for implementations in special purpose hardware. We show that the p-delta rule also implements gradient descent-with regard to a suitable error measure-although it does not require to compute derivatives. Furthermore it is shown through experiments on common real-world benchmark datasets that its performance is competitive with that of other learning approaches from neural networks and machine learning. It has recently been shown [Anthony, M. (2007). On the generalization error of fixed combinations of classifiers. Journal of Computer and System Sciences 73(5), 725-734; Anthony, M. (2004). On learning a function of perceptrons. In Proceedings of the 2004 IEEE international joint conference on neural networks (pp. 967-972): Vol. 2] that one can also prove quite satisfactory bounds for the generalization error of this new learning rule.

Equal Interval Range Approximation and Expanding Learning Rule for Multi-Layer Perceptrons and Applications

ECG Authentication Method Based on Parallel Multi-Scale One-Dimensional Residual Network with Center and Margin Loss.

EEG Recognition Based on Multiple Types of Information by Using Wavelet Packet Transform and Neural Networks

A learning rule for very simple universal approximators consisting of a single layer of perceptrons

A Multilayer Complex Neural Network Training Algorithm and Its Application in Adaptive Equalization

A SYMMETRIC TERNARY LOGIC BASED MULTI-VALUED LEARNING NETWORK

Application Of Precise-Spike-Driven Rule In Spiking Neural Networks For Optical Character Recognition

Generalised Perceptron Learning

Spatio-temporal MLP network for seizure prediction using EEG signals

Learning EEG topographical representation for classification via convolutional neural network

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity

Multistrategy Learning Using Genetic Algorithms and Neural Networks for Pattern Classification

The Improved Rapid Convergence Algorithm of the Connecting Rights in the BP Network

A gradient-based automatic optimization CNN framework for EEG state recognition

PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons

Attention model of EEG signals based on reinforcement learning

Bespoke Approximation of Multiplication-Accumulation and Activation Targeting Printed Multilayer Perceptrons

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

A robust learning algorithm for the extended AM neural network

Automatic electrocardiogram detection and classification using bidirectional long short-term memory network improved by Bayesian optimization

A Quick Learning Rule to Expand Stable Attraction Basins in Auto-Associative Neural Networks