Training neuron models with the Informax principle

Jianfeng Feng

DOI: https://doi.org/10.1016/S0925-2312(02)00368-5

IF: 6

2002-01-01

Neurocomputing

Abstract:In terms of the Informax principle, and the input–output relationship of the integrate-and-fire (IF) model, IF neuron learning rules are developed and applied to blind separation tasks.

What problem does this paper attempt to address?

Training integrate-and-fire neurons with the Informax principle II

Jianfeng Feng,Sun, Y.,Buxton, H.,Gang Wei

DOI: https://doi.org/10.1109/TNN.2003.809419

2003-01-01

Abstract:For pt I see J. Phys. A, vol. 35, p. 2379-94 (2002).We develop neuron learning rules using the Informax principle together with the input-output relationship of the integrate-and-fire (IF) model with Poisson inputs. The learning rule is then tested with constant inputs, time-varying inputs and images. For constant inputs, it is found that, under the Informax principle, a network of IF models with initially all positive weights tends to disconnect some connections between neurons. For time-varying inputs and images, we perform signal separation tasks called independent component analysis. Numerical simulations indicate that some number of inhibitory inputs improves the performance of the system in both biological and engineering senses.
Training the Integrate-and-fire Model with the Informax Principle: I

JF Feng,H Buxton,YC Deng

DOI: https://doi.org/10.1088/0305-4470/35/10/304

2002-01-01

Journal of Physics A Mathematical and General

Abstract:In terms of the informax principle, and the input-output relationship of the integrate-and-fire (IF) model, IF neuron learning rules are developed. For supervised learning and with uniform weight of synapses (the theoretically tractable case), we show that the derived learning rule is stable and the stable state is unique. For unsupervised learning, within physiologically reasonable parameter regions, both long-term potentiation (LTP) and long-term depression (LTD) could happen when the inhibitory input is weak, but LTD cannot be observed when inhibitory input is strong enough. When both LTP and LTD occur, LTD is observable when the output of the postsynaptic neuron is faster than pre-synaptic inputs, otherwise LTP is observable, as observed in recent experiments. Learning rules of general cases are also studied and numerical examples show that the derived learning rule tends to equalize the contribution of different inputs to the output firing rates.
A modelling study on discrimination tasks.

Jianfeng Feng,Feng Liu

DOI: https://doi.org/10.1016/S0303-2647(02)00064-3

IF: 1.957

2002-01-01

Biosystems

Abstract:We investigate the discrimination capacity of a single neuron model including the integrate-and-fire (IF) model and the IF-FHN model [Neural Networks, 14 (2001) 955], both theoretically and numerically. Both magnitude and frequency discrimination tasks are considered. It is found that for the magnitude discrimination task, with a small fraction of coherent inputs, an IF model and an IF-FHN model with inhibitory inputs can tell one input from the other. The total probability of misclassifications (TPM) is considerably reduced with the increasing of inhibitory inputs. However, the IF model and the IF-FHN model are incapable of performing the frequency discrimination task.
Training Spiking Neuronal Networks with Applications in Engineering Tasks.

Phill Rowcliffe,Jianfeng Feng

DOI: https://doi.org/10.1109/tnn.2008.2000999

2008-01-01

IEEE Transactions on Neural Networks

Abstract:In this paper, spiking neuronal models employing means, variances, and correlations for computation are introduced. We present two approaches in the design of spiking neuronal networks, both of which are applied to engineering tasks. In exploring the input-output relationship of integrate-and-fire (IF) neurons with Poisson inputs, we are able to define mathematically robust learning rules, which can be applied to multilayer and time-series networks. We show through experimental applications that it is possible to train spike-rate networks on function approximation problems and on the dynamic task of robot arm control.
End-to-end neural system identification with neural information flow

K. Seeliger,L. Ambrogioni,Y. Güçlütürk,L. M. van den Bulk,U. Güçlü,M. A. J. van Gerven

DOI: https://doi.org/10.1371/journal.pcbi.1008558

2021-02-04

PLoS Computational Biology

Abstract:Neural information flow (NIF) provides a novel approach for system identification in neuroscience. It models the neural computations in multiple brain regions and can be trained end-to-end via stochastic gradient descent from noninvasive data. NIF models represent neural information processing via a network of coupled tensors, each encoding the representation of the sensory input contained in a brain region. The elements of these tensors can be interpreted as cortical columns whose activity encodes the presence of a specific feature in a spatiotemporal location. Each tensor is coupled to the measured data specific to a brain region via low-rank observation models that can be decomposed into the spatial, temporal and feature receptive fields of a localized neuronal population. Both these observation models and the convolutional weights defining the information processing within regions are learned end-to-end by predicting the neural signal during sensory stimulation. We trained a NIF model on the activity of early visual areas using a large-scale fMRI dataset recorded in a single participant. We show that we can recover plausible visual representations and population receptive fields that are consistent with empirical findings.

biochemical research methods,mathematical & computational biology
A neuronet approach to information fusion.

Thomas S. Huang,Hess, C.P.,Hao Pan,Zhi-Pei Liang

DOI: https://doi.org/10.1109/MMSP.1997.602611

1997-01-01

Abstract:Neuronet approaches offer a unique and powerful tool for nonlinear information fusion. Unlike traditional techniques, neuronets do not require explicit environmental models or descriptions of sensor characteristics. This paper describes a technique for sensor fusion which makes use of a new neural model to combine data autonomously extracted from different sources. Application of the technique to bimodal recognition of combined speech/image signals is discussed.
A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Qiang Yu,Huajin Tang,Kay Chen Tan,Haoyong Yu

DOI: https://doi.org/10.1016/j.neucom.2013.06.052

IF: 6

2014-01-01

Neurocomputing

Abstract:Neural coding and learning are important components in cognitive memory system, by processing the sensory inputs and distinguishing different patterns to allow for higher level brain functions such as memory storage and retrieval. Benefitting from biological relevance, this paper presents a spiking neural network of leaky integrate-and-fire (LIF) neurons for pattern recognition. A biologically plausible supervised synaptic learning rule is used so that neurons can efficiently make a decision. The whole system contains encoding, learning and readout. Utilizing the temporal coding and learning, networks of spiking neurons can effectively and efficiently perform various classification tasks. It can classify complex patterns of activities stored in a vector, as well as the real-world stimuli. Our approach is also benchmarked on the nonlinearly separable Iris dataset. The proposed approach achieves a good generalization, with a classification accuracy of 99.63% for training and 92.55% for testing. In addition, the trained networks demonstrate that the temporal coding is a viable means for fast neural information processing.
Complex-Valued Recurrent Neural Network with IIR Neuron Model: Training and Applications

Chunguang Li,Xiaofeng Liao,Juebang Yu

DOI: https://doi.org/10.1007/s00034-002-0119-8

2002-01-01

Abstract:In this paper, based on the digital filter theory and approach,a new algorithm for training a complex-valued recurrent neuralnetwork, is proposed. Each recurrent neuron is modeled as aninfinite impulse response (IIR) filter. The network weights are updated by optimizing the IIR filter coefficients, and the optimization is based on thelayer-by-layer optimizing procedure as well as therecursive least-squares method. The performance of theproposed algorithm is demonstrated with application to a complex communication channel equalization. Our approachprovides a new way to perform fast training of complex-valuedrecurrent neural networks.
Research on the Method of Neural Network Modeling Based on FCM Algorithm and Its Application on Vision-based Sensors

Dibo Hou,Guangxin Zhang,Zekui Zhou

DOI: https://doi.org/10.2529/piers050108134357

2005-01-01

PIERS Online

Abstract:This paper proposes a novel fuzzy neural network model based on fuzzy clustering method. The model can accept continuous and discrete inputs together; the discrete input to the model is divided into several clusters by using fuzzy c-mean clustering algorithm (FCM). A fuzzy clustering neuron (FC-neuron) is designed to calculate a membership degree value belonging to one cluster for each discrete input. A four-layer hybrid neural network is constructed; fuzzy-neurons and FC-neurons construct the antecedent part of fuzzy rules. A multi-input multioutput hybrid neural network was designed by the novel modeling method and applied to vision-based sensors. Simulation results show this method is superior to the traditional neural network model in vision-based sensors.
Population Approach to a Neural Discrimination Task

Benoit Gaillard,Hilary Buxton,Jianfeng Feng

DOI: https://doi.org/10.1007/s00422-005-0039-3

IF: 3.072

2005-01-01

Biological Cybernetics

Abstract:This article gives insights into the possible neuronal processes involved in visual discrimination. We study the performance of a spiking network of Integrate-and-Fire (IF) neurons when performing a benchmark discrimination task. The task we adopted consists of determining the direction of moving dots in a noisy context using similar stimuli to those in the experiments of Newsome and colleagues. We present a neural model that performs the discrimination involved in this task. By varying the synaptic parameters of the IF neurons, we illustrate the counter-intuitive importance of the second-order statistics (input noise) in improving the discrimination accuracy of the model. We show that measuring the Firing Rate (FR) over a population enables the model to discriminate in realistic times, and even surprisingly significantly increases its discrimination accuracy over the single neuron case, despite the faster processing. We also show that increasing the input noise increases the discrimination accuracy but only at the expense of the speed at which we can read out the FR.
Neuronal Models with Masked Signals

Jianfeng Feng

2002-01-01

Abstract:Suppose that a neuron receive two signals masked by random noise. We ask after neuronal transformation whether the masked signals will become more separable or more mixed. We prove that with positive correlations between input singles, the output will be more separable. The critical value at which the total probability of misclassification is zero is analytically obtained.
What should a neuron aim for? Designing local objective functions based on information theory

Andreas C. Schneider,Valentin Neuhaus,David A. Ehrlich,Abdullah Makkeh,Alexander S. Ecker,Viola Priesemann,Michael Wibral

2024-12-03

Abstract:In modern deep neural networks, the learning dynamics of the individual neurons is often obscure, as the networks are trained via global optimization. Conversely, biological systems build on self-organized, local learning, achieving robustness and efficiency with limited global information. We here show how self-organization between individual artificial neurons can be achieved by designing abstract bio-inspired local learning goals. These goals are parameterized using a recent extension of information theory, Partial Information Decomposition (PID), which decomposes the information that a set of information sources holds about an outcome into unique, redundant and synergistic contributions. Our framework enables neurons to locally shape the integration of information from various input classes, i.e. feedforward, feedback, and lateral, by selecting which of the three inputs should contribute uniquely, redundantly or synergistically to the output. This selection is expressed as a weighted sum of PID terms, which, for a given problem, can be directly derived from intuitive reasoning or via numerical optimization, offering a window into understanding task-relevant local information processing. Achieving neuron-level interpretability while enabling strong performance using local learning, our work advances a principled information-theoretic foundation for local learning strategies.

Information Theory,Machine Learning,Neural and Evolutionary Computing
Reconstruction of Visual Image from Functional Magnetic Resonance Imaging Using Spiking Neuron Model

Yongqiang Ma,Hao Wu,Mengjiao Zhu,Pengju Ren,Nanning Zheng,Badong Chen

DOI: https://doi.org/10.1109/tcds.2017.2764948

IF: 4.546

2017-01-01

IEEE Transactions on Cognitive and Developmental Systems

Abstract:Spikes are the basic elements of information dissemination in the organic brain. A spike train, a series of discrete action potentials, incorporates the conception of time. Nowadays, the majority of existing artificial neural networks use numerical information to realize machine learning and cognition tasks. The advantage of numerical computation is its precision and remarkable performance on well-structured problems. However, cognition tasks have semi-structured or ill-structured information, even worse, some cannot reach a conclusion. According to biological experiments, the spiking neuron model is potentially more suitable for dealing with undetermined and unstructured problems. Researchers are trying to decode functional magnetic resonance imaging (fMRI) data, and this is a typical task to extract meaningful information from relatively undetermined and unstructured data. In this paper, we selected the Tempotron neuron model to analyze and reconstruct visual images from fMRI data when subjects received several kinds of visual stimuli; the preliminary results of pattern reconstruction were unpolished, but somewhat effective. Thus, a new structure for the spiking neural network was built to achieve better results. Several important aspects of the proposed model were discussed in this paper: the encoding of external stimulating signals, the extraction of effective features, the relative position of spike trains on the timeline, the back propagation of error, and the rationality of parameter selection. These aspects are crucial in the spiking neuron model’s implementation, and are worth further investigation in future studies.
On Model-guided Neural Networks for System Identification

Lei Lu,Ying Tan,Denny Oetomo,Iven Mareels,Erying Zhao,Shi An

DOI: https://doi.org/10.1109/SSCI44817.2019.9002703

2019-01-01

Abstract:Many techniques exist to train neural networks to approximate a complex systems, such as deep learning methods. It is well known that despite their robustness with respect to over-fitting, such trained models may be brittle as some fundamental principles of the systems are missing. For many engineering applications, the model class may be derived from first principles (or fundamental principles). In this paper, ideas from both methodologies are combined to arrive at a robust model that is interpretable from first principles, but goes beyond this by capturing structure from the available data. The paper presents two examples to illustrate the ideas. First a synthetic data set based on simulations is used. Next a well known data set from functional magnetic resonance imaging (fMRI) is used. In these two examples, a few representative neural networks are used in combination with model information coming from first principles. The preliminary results show that the framework is highly beneficial and yields excellent system identification fidelity.
IC Neuron: an Efficient Unit to Construct Neural Networks

Junyi An,Fengshan Liu,Furao Shen,Jian Zhao,Ruotong Li,Kepan Gao

DOI: https://doi.org/10.1016/j.neunet.2021.10.005

IF: 7.8

2021-01-01

Neural Networks

Abstract:As a popular machine learning method, neural networks can be used to solve many complex tasks. Their strong generalization ability comes from the representation ability of the basic neuron model. The most popular neuron is the MP neuron, which uses a linear transformation and a non-linear activation function to process the input successively. Inspired by the elastic collision model in physics, we propose a new neuron model that can represent more complex distributions. We term it Inter-layer collision (IC) neuron. The IC neuron divides the input space into multiple subspaces used to represent different linear transformations. This operation enhanced non-linear representation ability and emphasizes some useful input features for the given task. We build the IC networks by integrating the IC neurons into the fully-connected (FC), convolutional, and recurrent structures. The IC networks outperform the traditional networks in a wide range of experiments. We believe that the IC neuron can be a basic unit to build network structures.
Decoding Input Signals in Time Domain—A Model Approach

Jianfeng Feng,David Brown

DOI: https://doi.org/10.1023/b:jcns.0000025687.78916.5a

IF: 1.4529

2004-01-01

Journal of Computational Neuroscience

Abstract:From an observation of efferent interspike intervals of a neuron, we consider how to decode the input temporal information. It is found that the integrate-and-fire model is blind in the temporal domain due to the fact that its efferent firing rate is independent of the input temporal frequency. The conclusion is then confirmed for the integrate-and-fire model with correlated inputs, with reversal potentials, with a nonlinear leakage and with a subthreshold oscillation. For the Hodgkin-Huxley model, however, in terms of efferent firing rates alone, it is possible to read out the input temporal information.
Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders

Mathias Vast,Basile Van Cooten,Laure Soulier,Benjamin Piwowarski

DOI: https://doi.org/10.1145/3664190.3672528

2024-07-05

Abstract:With the recent addition of Retrieval-Augmented Generation (RAG), the scope and importance of Information Retrieval (IR) has expanded. As a result, the importance of a deeper understanding of IR models also increases. However, interpretability in IR remains under-explored, especially when it comes to the models' inner mechanisms. In this paper, we explore the possibility of adapting Integrated Gradient-based methods in an IR context to identify the role of individual neurons within the model. In particular, we provide new insights into the role of what we call "relevance" neurons, as well as how they deal with unseen data. Finally, we carry out an in-depth pruning study to validate our findings.

Information Retrieval
Morphological classification of neurons based on Sugeno fuzzy integration and multi-classifier fusion

Fuyun He,Guanglian Li,Haixing Song

DOI: https://doi.org/10.1038/s41598-024-66797-1

IF: 4.6

2024-07-12

Scientific Reports

Abstract:In order to extract more important morphological features of neuron images and achieve accurate classification of the neuron type, a method is proposed that uses Sugeno fuzzy integral integration of three optimized deep learning models, namely AlexNet, VGG11_bn, and ResNet-50. Firstly, using the pre-trained model of AlexNet and the output layer is fine-tuned to improve the model's performance. Secondly, in the VGG11_bn network, Global Average Pooling (GAP) is adopted to replace the traditional fully connected layer to reduce the number of parameters. Additionally, the generalization ability of the model is improved by transfer learning. Thirdly, the SE(squeeze and excitation) module is added to the ResNet-50 variant ResNeXt-50 to adjust the channel weight and capture the key information of the input data. The GELU activation function is used to better fit the data distribution. Finally, Sugeno fuzzy integral is used to fuse the output of each model to get the final classification result. The experimental results showed that on the Img_raw, Img_resample and Img_XYalign dataset, the accuracy of 4-category classification reached 98.04%, 91.75% and 93.13%, respectively, and the accuracy of 12-category classification reached 97.82%, 85.68% and 87.60%, respectively. The proposed method has good classification performance in the morphological classification of neurons.

multidisciplinary sciences
Spiking Perceptrons

P. Rowcliffe,Jianfeng Feng,H. Buxton

DOI: https://doi.org/10.1109/tnn.2006.873274

2006-01-01

IEEE Transactions on Neural Networks

Abstract:A more plausible biological version of the traditional perceptron is presented here with a learning rule which enables training of the neuron on nonlinear tasks. Three different models are introduced with varying inhibitory and excitatory synaptic connections. Using the derived learning rule, a single neuron is trained to successfully classify the XOR problem.
A Model of Invariant Object Recognition in the Visual System: Learning Rules, Activation Functions, Lateral Inhibition, and Information-Based Performance Measures

ET Rolls,T Milward

DOI: https://doi.org/10.1162/089976600300014845

IF: 3.278

2000-01-01

Neural Computation

Abstract:VisNet2 is a model to investigate some aspects of invariant visual object recognition in the primate visual system. It is a four-layer feedforward network with convergence to each part of a layer from a small region of the preceding layer, with competition between the neurons within a layer and with a trace learning rule to help it learn transform invariance. The trace rule is a modified Hebbian rule, which modifies synaptic weights according to both the current firing rates and the firing rates to recently seen stimuli. This enables neurons to learn to respond similarly to the gradually transforming inputs it receives, which over the short term are likely to be about the same object, given the statistics of normal visual inputs. First, we introduce for VisNet2 both single-neuron and multiple-neuron information-theoretic measures of its ability to respond to transformed stimuli. Second, using these measures, we show that quantitatively resetting the trace between stimuli is not necessary for good performance. Third, it is shown that the sigmoid activation functions used in VisNet2, which allow the sparseness of the representation to be controlled, allow good performance when using sparse distributed representations. Fourth, it is shown that VisNet2 operates well with medium-range lateral inhibition with a radius in the same order of size as the region of the preceding layer from which neurons receive inputs. Fifth, in an investigation of different learning rules for learning transform invariance, it is shown that VisNet2 operates better with a trace rule that incorporates in the trace only activity from the preceding presentations of a given stimulus, with no contribution to the trace from the current presentation, and that this is related to temporal difference learning.

Training neuron models with the Informax principle

Training integrate-and-fire neurons with the Informax principle II

Training the Integrate-and-fire Model with the Informax Principle: I

A modelling study on discrimination tasks.

Training Spiking Neuronal Networks with Applications in Engineering Tasks.

End-to-end neural system identification with neural information flow

A neuronet approach to information fusion.

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Complex-Valued Recurrent Neural Network with IIR Neuron Model: Training and Applications

Research on the Method of Neural Network Modeling Based on FCM Algorithm and Its Application on Vision-based Sensors

Population Approach to a Neural Discrimination Task

Neuronal Models with Masked Signals

What should a neuron aim for? Designing local objective functions based on information theory

Reconstruction of Visual Image from Functional Magnetic Resonance Imaging Using Spiking Neuron Model

On Model-guided Neural Networks for System Identification

IC Neuron: an Efficient Unit to Construct Neural Networks

Decoding Input Signals in Time Domain—A Model Approach

Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders

Morphological classification of neurons based on Sugeno fuzzy integration and multi-classifier fusion

Spiking Perceptrons

A Model of Invariant Object Recognition in the Visual System: Learning Rules, Activation Functions, Lateral Inhibition, and Information-Based Performance Measures