Learning in A Higher-Order Simple Perceptron

JF Feng,B Tirozzi

DOI: https://doi.org/10.1016/s0895-7177(99)00191-0

1999-01-01

Mathematical and Computer Modelling

Abstract:We find a sufficient condition for the convergence of the learning process of the higher order simple perceptron.

What problem does this paper attempt to address?

A Discussion of the Learning in a Simple Perceptron

Jianfeng Feng

1995-01-01

Abstract:In this paper,we extend the convergence of the simple perceptron learning rule to the case that the set of inputs is infinity or a region。 When the set of inputs is linearly separable,we prove that a simple perceptron always improves its performance。As the set of the inputs is ‘strong’linearly separable,then within finite time the connections among units converge to a limit which separates the inputs.The convergence rate is also estimated。
Generalization Errors of the Simple Perceptron

JF Feng

DOI: https://doi.org/10.1088/0305-4470/31/17/013

1998-01-01

Abstract:To find an exact form for the generalization error of a learning machine is an open problem, even in the simplest case: simple perceptron learning. We introduce a new approach to tackle the problem. The generalization error of the simple perceptron is expressed as a linear combination of extreme values of inputs. With the help of extreme value theory in statistics we then obtain an exact form of the generalization error of the simple perceptron in the case of the worst learning. Generalization errors of the higher-order perceptron taking the form of an inverse power law in the number of examples are also considered.
Finite convergence of a fuzzy δ rule for a fuzzy perceptron

Long Li,Jie Yang,Yan Liu,Wei Wu

2008-01-01

Neural Network World

Abstract:This paper considers a fuzzy perceptron that has the same topological structure as the conventional linear perceptron. A learning algorithm based on a fuzzy S rule is proposed for this fuzzy perceptron. The inner operations involved in the working process of this fuzzy perceptron are based on the max-min logical operations rather than conventional multiplication and summation, etc. The initial values of the network weights are fixed as 1. It is shown that each network weight is non-increasing in the training process and remains unchanged once it is less than 0.5. The learning algorithm has an advantage, as proved in this paper, that it converges in a finite number of steps if the training patterns are fuzzily separable. This result generalizes a corresponding classical result for conventional linear perceptrons. Some numerical experiments for the learning algorithm are provided to support our theoretical findings. © ICS AS CR 2008.
FINITE CONVERGENCE OF A FUZZY delta RULE FOR A FUZZY PERCEPTRON

Long Li,Jie Yang,Yan Liu,Wei Wu

2008-01-01

Neural Network World

Abstract:This paper considers a fuzzy perceptron that has the same topological structure as the conventional linear perceptron. A learning algorithm based on a fuzzy 6 rule is proposed for this fuzzy perceptron. The inner operations involved in the working process of this fuzzy perceptron are based on the max-min logical operations rather than conventional multiplication and summation, etc. The initial values of the network weights are fixed as 1. It is shown that each network weight is non-increasing in the training process and remains unchanged once it is less than 0.5. The learning algorithm has an advantage, as proved in this paper, that it converges in a finite number of steps if the training patterns are fuzzily separable. This result generalizes a corresponding classical result for conventional linear perceptrons. Some numerical experiments for the learning algorithm are provided to support our theoretical findings.
Finite Convergence for Recurrent Fuzzy Perceptron

Long Li

2011-01-01

Abstract:The main function of fuzzy perceptron is to discriminate which categories the samples are in by weight learning.An algorithm for a recurrent fuzzy perceptron based on fuzzy logic is presented,and the network structure of the recurrent fuzzy perceptron is similar to traditional perceptron based on addition-production,and the dynamic recursion term is added.Initial weights of network are set to be constant zero,in the case where the dimension of the input vectors is two and the training examples are separable,its finite convergence is proved,i.e.,the training procedure for the network weights will stop in finite steps,and when the dimension is greater than two,stronger conditions are needed to guarantee the finite convergence.
A new training algorithm for a fuzzy perceptron and its convergence

Jie Yang,Wei Wu,Zhiqiong Shao

DOI: https://doi.org/10.1007/11427391_97

2005-01-01

Abstract:In this paper, we present a new training algorithm for a fuzzy perceptron. In the case where the dimension of the input vectors is two and the training examples are separable, we can prove a finite convergence, i.e., the training procedure for the network weights will stop after finite steps. When the dimension is greater than two, stronger conditions are needed to guarantee the finite convergence.
On-line learning through simple perceptron with a margin

Kazuyuki Hara,Masato Okada

DOI: https://doi.org/10.48550/arXiv.cond-mat/0306150

2003-06-05

Disordered Systems and Neural Networks

Abstract:We analyze a learning method that uses a margin $\kappa$ {\it a la} Gardner for simple perceptron learning. This method corresponds to the perceptron learning when $\kappa=0$, and to the Hebbian learning when $\kappa \to \infty$. Nevertheless, we found that the generalization ability of the method was superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning curve of this method through computer simulation and found that it was the same as for perceptron learning. We also investigated an adaptive margin control method.
Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

Kazuyuki Hara,Seiji Miyoshi

DOI: https://doi.org/10.1143/JPSJ.81.064002

2016-08-24

Abstract:In ensemble teacher learning, ensemble teachers have only uncertain information about the true teacher, and this information is given by an ensemble consisting of an infinite number of ensemble teachers whose variety is sufficiently rich. In this learning, a student learns from an ensemble teacher that is iteratively selected randomly from a pool of many ensemble teachers. An interesting point of ensemble teacher learning is the asymptotic behavior of the student to approach the true teacher by learning from ensemble teachers. The student performance is improved by using the Hebbian learning rule in the learning. However, the perceptron learning rule cannot improve the student performance. On the other hand, we proposed a perceptron learning rule with a margin. This learning rule is identical to the perceptron learning rule when the margin is zero and identical to the Hebbian learning rule when the margin is infinity. Thus, this rule connects the perceptron learning rule and the Hebbian learning rule continuously through the size of the margin. Using this rule, we study changes in the learning behavior from the perceptron learning rule to the Hebbian learning rule by considering several margin sizes. From the results, we show that by setting a margin of kappa > 0, the effect of an ensemble appears and becomes significant when a larger margin kappa is used.

Disordered Systems and Neural Networks
Finite Size Scaling of Perceptron

Elka Korutcheva,N. Tonchev,N.Tonchev

DOI: https://doi.org/10.48550/arXiv.cond-mat/0001100

2000-01-10

Statistical Mechanics

Abstract:We study the first-order transition in the model of a simple perceptron with continuous weights and large, bit finite value of the inputs. Making the analogy with the usual finite-size physical systems, we calculate the shift and the rounding exponents near the transition point. In the case of a general perceptron with larger variety of inputs, the analysis only gives bounds for the exponents.
Ensemble learning of linear perceptron; Online learning theory

Kazuyuki Hara,Masato Okada

DOI: https://doi.org/10.48550/arXiv.cond-mat/0402069

2004-02-03

Disordered Systems and Neural Networks

Abstract:Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous or inhomogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows. For learning with homogeneous initial weight vectors, the generalization error using an infinite number of linear student perceptrons is equal to only half that of a single linear perceptron, and converges with that of the infinite case with O(1/K) for a finite number of K linear perceptrons. For learning with inhomogeneous initial weight vectors, it is advantageous to use an approach of weighted averaging over the output of the linear perceptrons, and we show the conditions under which the optimal weights are constant during the learning process. The optimal weights depend on only correlation of the initial weight vectors.
Convergence Theorems for a Class of Learning Algorithms with VLRPs

JF Feng,B Tirozzi

DOI: https://doi.org/10.1016/s0925-2312(96)00043-4

IF: 6

1997-01-01

Neurocomputing

Abstract:We first consider the convergence of the simple competitive learning with vanishing learning rate parameters (VLRPs). Examples show that even in this setting the learning fails to converge in general. This brings us to consider the following problem: to find out a family of VLRPs such that an algorithm with the VLRPs reaches the global minima with probability one. Here, we present an approach different from stochastic approximation theory and determine a new family of VLRPs such that the corresponding learning algorithm gets out of the metastable states with probability one. In the literature it is generally believed that a family of reasonable VLRPs is of the order of 1tα for 12 < α ≤ 1, where t is the time. However, we find that a family of VLRPs which makes the algorithm go to the global minima should be between 1log t and 1√log t.
A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

Chunguang Li,Xiaofeng Liao,Zhongfu Wu,Juebang Yu

DOI: https://doi.org/10.3969/j.issn.1003-0530.2000.04.009

IF: 4.729

2000-01-01

Signal Processing

Abstract:In this brief paper, a Complex Hebbian-type learning algorithm for training a complex neural net work whose inputs, outputs and weights are all complex was proposed. We derive thealgorithm in TLS sense other than in LS or LMS sense. The performance of the proposed algorithm is demonstrated with application in complex adaptive IIR filtering
Generalization ability of a perceptron with non-monotonic transfer function

Jun-ichi Inoue,Hidetoshi Nishimori,Yoshiyuki Kabashima

DOI: https://doi.org/10.1103/PhysRevE.58.849

1998-01-29

Abstract:We investigate the generalization ability of a perceptron with non-monotonic transfer function of a reversed-wedge type in on-line mode. This network is identical to a parity machine, a multilayer network. We consider several learning algorithms. By the perceptron algorithm the generalization error is shown to decrease by the ${\alpha}^{-1/3}$-law similarly to the case of a simple perceptron in a restricted range of the parameter $a$ characterizing the non-monotonic transfer function. For other values of $a$, the perceptron algorithm leads to the state where the weight vector of the student is just opposite to that of the teacher. The Hebbian learning algorithm has a similar property; it works only in a limited range of the parameter. The conventional AdaTron algorithm does not give a vanishing generalization error for any values of $a$.We thus introduce a modified AdaTron algorithm which yields a good performance for all values of $a$. We also investigate the effects of optimization of the learning rate as well as of the learning algorithm. Both methods give excellent learning curves proportional to ${\alpha}^{-1}$. The latter optimization is related to the Bayes statistics and is shown to yield useful hints to extract maximum amount of information necessary to accelerate learning processes.

Disordered Systems and Neural Networks
On-line learning and generalisation in coupled perceptrons

D. Bolle',P. Kozlowski

DOI: https://doi.org/10.48550/arXiv.cond-mat/0111493

2001-11-26

Disordered Systems and Neural Networks

Abstract:We study supervised learning and generalisation in coupled perceptrons trained on-line using two learning scenarios. In the first scenario the teacher and the student are independent networks and both are represented by an Ashkin-Teller perceptron. In the second scenario the student and the teacher are simple perceptrons but are coupled by an Ashkin-Teller type four-neuron interaction term. Expressions for the generalisation error and the learning curves are derived for various learning algorithms. The analytic results find excellent confirmation in numerical simulations.
The Concept of Forward-Forward Learning Applied to a Multi Output Perceptron

K. Fredrik Karlsson

2023-04-07

Abstract:The concept of a recently proposed Forward-Forward learning algorithm for fully connected artificial neural networks is applied to a single multi output perceptron for classification. The parameters of the system are trained with respect to increased (decreased) "goodness" for correctly (incorrectly) labelled input samples. Basic numerical tests demonstrate that the trained perceptron effectively deals with data sets that have non-linear decision boundaries. Moreover, the overall performance is comparable to more complex neural networks with hidden layers. The benefit of the approach presented here is that it only involves a single matrix multiplication.

Machine Learning,Neural and Evolutionary Computing
Spiking Perceptrons

P. Rowcliffe,Jianfeng Feng,H. Buxton

DOI: https://doi.org/10.1109/tnn.2006.873274

2006-01-01

IEEE Transactions on Neural Networks

Abstract:A more plausible biological version of the traditional perceptron is presented here with a learning rule which enables training of the neuron on nonlinear tasks. Three different models are introduced with varying inhibitory and excitatory synaptic connections. Using the derived learning rule, a single neuron is trained to successfully classify the XOR problem.
Learning-induced Pattern Classification in a Chaotic Neural Network

Yang Li,Ping Zhu,Xiaoping Xie,Guoguang He,Kazuyuki Aihara

DOI: https://doi.org/10.1016/j.physleta.2011.10.062

IF: 2.707

2012-01-01

Physics Letters A

Abstract:In this Letter, we propose a Hebbian learning rule with passive forgetting (HLRPF) for use in a chaotic neural network (CNN). We then define the indices based on the Euclidean distance to investigate the evolution of the weights in a simplified way. Numerical simulations demonstrate that, under suitable external stimulations, the CNN with the proposed HLRPF acts as a fuzzy-like pattern classifier that performs much better than an ordinary CNN. The results imply relationship between learning and recognition.
Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks

Marco Benedetti,Enrico Ventura,Enzo Marinari,Giancarlo Ruocco,Francesco Zamponi

DOI: https://doi.org/10.1063/5.0084219

2022-03-07

Abstract:The Hebbian unlearning algorithm, i.e. an unsupervised local procedure used to improve the retrieval properties in Hopfield-like neural networks, is numerically compared to a supervised algorithm to train a linear symmetric perceptron. We analyze the stability of the stored memories: basins of attraction obtained by the Hebbian unlearning technique are found to be comparable in size to those obtained in the symmetric perceptron, while the two algorithms are found to converge in the same region of Gardner's space of interactions, having followed similar learning paths. A geometric interpretation of Hebbian unlearning is proposed to explain its optimal performances. Because the Hopfield model is also a prototypical model of disordered magnetic system, it might be possible to translate our results to other models of interest for memory storage in materials.

Disordered Systems and Neural Networks,Statistical Mechanics,Biological Physics,Neurons and Cognition
Learning Algorithm for Fuzzy Perceptron with Max-Product Composition

Xin Yu,Mian Xie,Li Xia Tang,Chen Yu Li

DOI: https://doi.org/10.4028/www.scientific.net/amm.687-691.1359

2014-01-01

Applied Mechanics and Materials

Abstract:Fuzzy neural networks is a powerful computational model, which integrates fuzzy systems with neural networks, and fuzzy perceptron is a kind of this neural networks. In this paper, a learning algorithm is proposed for a fuzzy perceptron with max-product composition, and the topological structure of this fuzzy perceptron is the same as conventional linear perceptrons. The inner operations involved in the working process of this fuzzy perceptron are based on the max-product logical operations rather than conventional multiplication and summation etc. To illustrate the finite convergence of proposed algorithm, some numerical experiments are provided.
Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron

Christian Schmid,James M. Murray

2024-10-29

Abstract:The ability of a brain or a neural network to efficiently learn depends crucially on both the task structure and the learning rule. Previous works have analyzed the dynamical equations describing learning in the relatively simplified context of the perceptron under assumptions of a student-teacher framework or a linearized output. While these assumptions have facilitated theoretical understanding, they have precluded a detailed understanding of the roles of the nonlinearity and input-data distribution in determining the learning dynamics, limiting the applicability of the theories to real biological or artificial neural networks. Here, we use a stochastic-process approach to derive flow equations describing learning, applying this framework to the case of a nonlinear perceptron performing binary classification. We characterize the effects of the learning rule (supervised or reinforcement learning, SL/RL) and input-data distribution on the perceptron's learning curve and the forgetting curve as subsequent tasks are learned. In particular, we find that the input-data noise differently affects the learning speed under SL vs. RL, as well as determines how quickly learning of a task is overwritten by subsequent learning. Additionally, we verify our approach with real data using the MNIST dataset. This approach points a way toward analyzing learning dynamics for more-complex circuit architectures.

Machine Learning,Neurons and Cognition

Learning in A Higher-Order Simple Perceptron

A Discussion of the Learning in a Simple Perceptron

Generalization Errors of the Simple Perceptron

Finite convergence of a fuzzy δ rule for a fuzzy perceptron

FINITE CONVERGENCE OF A FUZZY delta RULE FOR A FUZZY PERCEPTRON

Finite Convergence for Recurrent Fuzzy Perceptron

A new training algorithm for a fuzzy perceptron and its convergence

On-line learning through simple perceptron with a margin

Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

Finite Size Scaling of Perceptron

Ensemble learning of linear perceptron; Online learning theory

Convergence Theorems for a Class of Learning Algorithms with VLRPs

A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

Generalization ability of a perceptron with non-monotonic transfer function

On-line learning and generalisation in coupled perceptrons

The Concept of Forward-Forward Learning Applied to a Multi Output Perceptron

Spiking Perceptrons

Learning-induced Pattern Classification in a Chaotic Neural Network

Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks

Learning Algorithm for Fuzzy Perceptron with Max-Product Composition

Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron