Methods for Estimating Neural Information

James V Stone

2022-12-20

Abstract:Estimating the Shannon information associated with individual neurons is a non-trivial problem. Three key methods used to estimate the mutual information between neuron inputs and outputs are described, and a list of further readings is provided.

Neurons and Cognition,Information Theory

What problem does this paper attempt to address?

Mutual information analysis on non-stationary neuron importance for brain machine interfaces.

Yuxi Liao,Yiwen Wang,Xiaoxiang Zheng,Jose C Principe

DOI: https://doi.org/10.1109/EMBC.2012.6346533

2012-01-01

Abstract:Decoding with the important neuron subset has been widely used in brain machine interfaces (BMIs), as an effective strategy to reduce computational complexity. Previous works usually assume stationary of neuron importance, which may not be true according to recent research. We propose to conduct a mutual information evaluation to track the time-varying neuron importance over time. We found worth noting changes both in information amount and space distribution in our experiment. When the method is applied with a Kalman filter, the decoding performance achieve is better (with higher correlation coefficient) than when a fixed subset, which shows that time-varying neuron importance should be considered in adaptive algorithms.
Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Wentao Huang,Kechen Zhang

DOI: https://doi.org/10.3390/e21030243

2019-03-05

Abstract:Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

Information Theory,Statistics Theory,Neurons and Cognition
InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization

Zhengyang Hu,Song Kang,Qunsong Zeng,Kaibin Huang,Yanchao Yang

2024-02-16

Abstract:Estimating mutual correlations between random variables or data streams is essential for intelligent behavior and decision-making. As a fundamental quantity for measuring statistical relationships, mutual information has been extensively studied and utilized for its generality and equitability. However, existing methods often lack the efficiency needed for real-time applications, such as test-time optimization of a neural network, or the differentiability required for end-to-end learning, like histograms. We introduce a neural network called InfoNet, which directly outputs mutual information estimations of data streams by leveraging the attention mechanism and the computational efficiency of deep learning infrastructures. By maximizing a dual formulation of mutual information through large-scale simulated training, our approach circumvents time-consuming test-time optimization and offers generalization ability. We evaluate the effectiveness and generalization of our proposed mutual information estimation scheme on various families of distributions and applications. Our results demonstrate that InfoNet and its training process provide a graceful efficiency-accuracy trade-off and order-preserving properties. We will make the code and models available as a comprehensive toolbox to facilitate studies in different fields requiring real-time mutual information estimation.

Information Theory
Neural Joint Entropy Estimation

Yuval Shalev,Amichai Painsky,Irad Ben-Gal

DOI: https://doi.org/10.1109/tnnls.2022.3204919

IF: 14.255

2022-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Correlation in Neuronal Calcium Spiking: Quantification based on Empirical Mutual Information Rate

Sathish Ande,Srinivas Avasarala,Ajith Karunarathne,Lopamudra Giri,Soumya Jana

DOI: https://doi.org/10.48550/arXiv.2105.03400

2021-05-07

Neurons and Cognition

Abstract:Quantification of neuronal correlations in neuron populations helps us to understand neural coding rules. Such quantification could also reveal how neurons encode information in normal and disease conditions like Alzheimer's and Parkinson's. While neurons communicate with each other by transmitting spikes, there would be a change in calcium concentration within the neurons inherently. Accordingly, there would be correlations in calcium spike trains and they could have heterogeneous memory structures. In this context, estimation of mutual information rate in calcium spike trains assumes primary significance. However, such estimation is difficult with available methods which would consider longer blocks for convergence without noticing that neuronal information changes in short time windows. Against this backdrop, we propose a faster method that exploits the memory structures in pair of calcium spike trains to quantify mutual information shared between them. Our method has shown superior performance with example Markov processes as well as experimental spike trains. Such mutual information rate analysis could be used to identify signatures of neuronal behavior in large populations in normal and abnormal conditions.
Rate of mutual information between coarse-grained non-Markovian variables

Andre Cardoso Barato,David Hartich,Udo Seifert

DOI: https://doi.org/10.1007/s10955-013-0834-5

2013-09-30

Abstract:The problem of calculating the rate of mutual information between two coarse-grained variables that together specify a continuous time Markov process is addressed. As a main obstacle, the coarse-grained variables are in general non-Markovian, therefore, an expression for their Shannon entropy rates in terms of the stationary probability distribution is not known. A numerical method to estimate the Shannon entropy rate of continuous time hidden-Markov processes from a single time series is developed. With this method the rate of mutual information can be determined numerically. Moreover, an analytical upper bound on the rate of mutual information is calculated for a class of Markov processes for which the transition rates have a bipartite character. Our general results are illustrated with explicit calculations for four-state networks.

Statistical Mechanics
Determination of relevant neuron-neuron connections for neural prosthetics using time-delayed mutual information: tutorial and preliminary results

Alexander Taghva,Dong Song,Robert E Hampson,Sam A Deadwyler,Theodore W Berger

DOI: https://doi.org/10.1016/j.wneu.2011.09.002

Abstract:Background: Identification of functional dependence among neurons is a necessary component in both the rational design of neural prostheses as well as in the characterization of network physiology. The objective of this article is to provide a tutorial for neurosurgeons regarding information theory, specifically time-delayed mutual information, and to compare time-delayed mutual information, an information theoretic quantity based on statistical dependence, with cross-correlation, a commonly used metric for this task in a preliminary analysis of rat hippocampal neurons. Methods: Spike trains were recorded from rats performing delayed nonmatch-to-sample task using an array of electrodes surgically implanted into the hippocampus of each hemisphere of the brain. In addition, spike train simulations of positively correlated neurons, negatively correlated neurons, and neurons correlated by nonlinear functions were generated. These were evaluated by time-delayed mutual information (MI) and cross-correlation. Results: Application of time-delayed MI to experimental data indicated the optimal bin size for information capture in the CA3-CA1 system was 40 ms, which may provide some insight into the spatiotemporal nature of encoding in the rat hippocampus. On simulated data, time-delayed MI showed peak values at appropriate time lags in positively correlated, negatively correlated, and complexly correlated data. Cross-correlation showed peak and troughs with positively correlated and negatively correlated data, but failed to capture some higher order correlations. Conclusions: Comparison of time-delayed MI to cross-correlation in identification of functionally dependent neurons indicates that the methods are not equivalent. Time-delayed MI appeared to capture some interactions between CA3-CA1 neurons at physiologically plausible time delays missed by cross-correlation. It should be considered as a method for identification of functional dependence between neurons and may be useful in the development of neural prosthetics.
Correlated Firing and the Information Represented by Neurons in Short Epochs

SR Schultz,S Panzeri,A Treves,ET Rolls

DOI: https://doi.org/10.1016/s0925-2312(99)00040-5

IF: 6

1999-01-01

Neurocomputing

Abstract:Is the information transmitted by an ensemble of neurons determined solely by the number of spikes fired by each cell, or do correlations in the emission of action potentials also play a significant role? We derive a simple formula which enables this question to be answered rigorously for short timescales. Using a power series expansion, the mutual information the ensemble of neurons conveys about external stimuli is broken down into firing rate and correlation components. This analysis leads to a new quantification procedure which is directly applicable to simultaneous multiple neuron recordings. It also shows rigorously that even when noise correlations are not stimulus modulated, synergistic effects can arise if signal and noise correlation are of opposite sign. (C) 1999 Elsevier Science B.V. All rights reserved.
Quantifying information stored in synaptic connections rather than in firing patterns of neural networks

Xinhao Fan,Shreesh P Mysore

2024-11-27

Abstract:A cornerstone of our understanding of both biological and artificial neural networks is that they store information in the strengths of connections among the constituent neurons. However, in contrast to the well-established theory for quantifying information encoded by the firing patterns of neural networks, little is known about quantifying information encoded by its synaptic connections. Here, we develop a theoretical framework using continuous Hopfield networks as an exemplar for associative neural networks, and data that follow mixtures of broadly applicable multivariate log-normal distributions. Specifically, we analytically derive the Shannon mutual information between the data and singletons, pairs, triplets, quadruplets, and arbitrary n-tuples of synaptic connections within the network. Our framework corroborates well-established insights about storage capacity of, and distributed coding by, neural firing patterns. Strikingly, it discovers synergistic interactions among synapses, revealing that the information encoded jointly by all the synapses exceeds the 'sum of its parts'. Taken together, this study introduces an interpretable framework for quantitatively understanding information storage in neural networks, one that illustrates the duality of synaptic connectivity and neural population activity in learning and memory.

Neurons and Cognition,Information Theory,Biological Physics
Estimating quantum mutual information through a quantum neural network

Myeongjin Shin,Junseo Lee,Kabgyun Jeong

DOI: https://doi.org/10.1007/s11128-023-04253-1

IF: 1.965

2024-02-11

Quantum Information Processing

Abstract:We propose a method of quantum machine learning called quantum mutual information neural estimation (QMINE) for estimating von Neumann entropy and quantum mutual information, which are fundamental properties in quantum information theory. The QMINE proposed here basically utilizes a technique of quantum neural networks (QNNs), to minimize a loss function that determines the von Neumann entropy, and thus quantum mutual information, which is believed more powerful to process quantum datasets than conventional neural networks due to quantum superposition and entanglement. To create a precise loss function, we propose a quantum Donsker-Varadhan representation (QDVR), which is a quantum analog of the classical Donsker-Varadhan representation. By exploiting a parameter shift rule on parameterized quantum circuits, we can efficiently implement and optimize the QNN and estimate the quantum entropies using the QMINE technique. Furthermore, numerical observations support our predictions of QDVR and demonstrate the good performance of QMINE.

physics, multidisciplinary,quantum science & technology, mathematical
Summary of Information Theoretic Quantities

Robin A. A. Ince,Stefano Panzeri,Simon R. Schultz

DOI: https://doi.org/10.1007/978-1-4614-7320-6_306-1

2015-01-08

Abstract:Information theory is a practical and theoretical framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. As a framework it has a number of useful properties: it provides a general measure sensitive to any relationship, not only linear effects; its quantities have meaningful units which in many cases allow direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single experimental trials, rather than in averages over multiple trials. A variety of information theoretic quantities are in common use in neuroscience - including the Shannon entropy, Kullback-Leibler divergence, and mutual information. In this entry, we introduce and define these quantities. Further details on how these quantities can be estimated in practice are provided in the entry "Estimation of Information-Theoretic Quantities" and examples of application of these techniques in neuroscience can be found in the entry "Applications of Information-Theoretic Quantities in Neuroscience".

Neurons and Cognition
Estimating mutual information for spike trains: a bird song example

Jake Witter,Conor Houghton

2023-09-14

Abstract:Zebra finch are a model animal used in the study of audition. They are adept at recognizing zebra finch songs and the neural pathway involved in song recognition is well studied. Here, this example is used to illustrate the estimation of mutual information between stimulus and response using a Kozachenko-Leonenko estimator. The challenge in calculating mutual information for spike trains is that there are no obvious coordinates for the data. The Kozachenko-Leonenko estimator does not require coordinates, it relies only on the distance between data points. In the case of bird song, estimating the mutual information demonstrates that the information content of spiking does not diminish as the song progresses.

Information Theory,Neurons and Cognition
Correlations and the Encoding of Information in the Nervous System

S Panzeri,SR Schultz,A Treves,ET Rolls

DOI: https://doi.org/10.1098/rspb.1999.0736

1999-01-01

Abstract:Is the information transmitted by an ensemble of neurons determined solely by the number of spikes fired by each cell, or do correlations in the emission of action potentials also play a significant role? We derive a simple formula which enables this question to be answered rigorously for short time-scales. The formula quantifies the corrections to the instantaneous information rate which result from correlations in spike emission between pairs of neurons. The mutual information that the ensemble of neurons conveys about external stimuli can thus be broken down into firing rate and correlation components. This analysis provides fundamental constraints upon the nature of information coding, showing that over short time-scales correlations cannot dominate information representation, that stimulus-independent correlations may lead to synergy (where the neurons together convey more information than they would if they were considered independently), but that only certain combinations of the different sources of correlation result in significant synergy rather than in redundancy or in negligible effects. This analysis leads to a new quantification procedure which is directly applicable to simultaneous multiple neuron recordings.
Does a larger neural network mean greater information transmission efficiency?

Bartosz Paprocki,Agnieszka Pregowska,Janusz Szczepanski

2023-07-25

Abstract:Realistic modeling of brain involves large number of neurons. The important question is how this size affects transmission efficiency? Here, these issue is studied in terms of Shannon's Theory. Mutual Information between input and output signals for simple class of networks with an increasing number of neurons is analyzed theoretically and numerically. Levy-Baxter neural model is applied. It turned out, that for these networks the Mutual Information converges, with increasing size, asymptotically very slowly to saturation level. This suggests that from certain level, the increase of neurons number does not imply significant increase in transmission efficiency, contributes rather to reliability.

Neurons and Cognition,Quantitative Methods
An Information Theoretic Approach to the Contributions of the Firing Rates and the Correlations Between the Firing of Neurons.

ET Rolls,L Franco,NC Aggelopoulos,S Reece

DOI: https://doi.org/10.1152/jn.01070.2002

2003-01-01

Journal of Neurophysiology

Abstract:To analyze the extent to which populations of neurons encode information in the numbers of spikes each neuron emits or in the relative time of firing of the different neurons that might reflect synchronization, we developed and analyzed the performance of an information theoretic approach. The formula quantifies the corrections to the instantaneous information rate that result from correlations in spike emission between pairs of neurons. We showed how these cross-cell terms can be separated from the correlations that occur between the spikes emitted by each neuron, the auto-cell terms in the information rate expansion. We also described a method to test whether the estimate of the amount of information contributed by stimulus-dependent synchronization is significant. With simulated data, we show that the approach can separate information arising from the number of spikes emitted by each neuron from the redundancy that can arise if neurons have common inputs and from the synergy that can arise if cells have stimulus-dependent synchronization. The usefulness of the approach is also demonstrated by showing how it helps to interpret the encoding shown by neurons in the primate inferior temporal visual cortex. When applied to a sample dataset of simultaneously recorded inferior temporal cortex neurons, the algorithm showed that most of the information is available in the number of spikes emitted by each cell; that there is typically just a small degree (approximately 12%) of redundancy between simultaneously recorded inferior temporal cortex (IT) neurons; and that there is very little gain of information that arises from stimulus-dependent synchronization effects in these neurons.
Quantifying evoked responses through information-theoretical measures

Julian Fuhrer,Kyrre Glette,Anaïs Llorens,Tor Endestad,Anne-Kristin Solbakk,Alejandro Omar Blenkmann

DOI: https://doi.org/10.3389/fninf.2023.1128866

2023-05-23

Abstract:Information theory is a viable candidate to advance our understanding of how the brain processes information generated in the internal or external environment. With its universal applicability, information theory enables the analysis of complex data sets, is free of requirements about the data structure, and can help infer the underlying brain mechanisms. Information-theoretical metrics such as Entropy or Mutual Information have been highly beneficial for analyzing neurophysiological recordings. However, a direct comparison of the performance of these methods with well-established metrics, such as the t-test, is rare. Here, such a comparison is carried out by evaluating the novel method of Encoded Information with Mutual Information, Gaussian Copula Mutual Information, Neural Frequency Tagging, and t-test. We do so by applying each method to event-related potentials and event-related activity in different frequency bands originating from intracranial electroencephalography recordings of humans and marmoset monkeys. Encoded Information is a novel procedure that assesses the similarity of brain responses across experimental conditions by compressing the respective signals. Such an information-based encoding is attractive whenever one is interested in detecting where in the brain condition effects are present.
Sampling bias corrections for accurate neural measures of redundant, unique, and synergistic information

Loren Kocillari,Gabriel Matias Lorenz,Nicola Marie Engel,Marco Celotto,Sebastiano Curreli,Simone Blanco Malerba,Andreas K Engel,Tommaso Fellin,Stefano Panzeri

DOI: https://doi.org/10.1101/2024.06.04.597303

2024-06-05

Abstract:Shannon Information theory has long been a tool of choice to measure empirically how populations of neurons in the brain encode information about cognitive variables. Recently, Partial Information Decomposition (PID) has emerged as principled way to break down this information into components identifying not only the unique information carried by each neuron, but also whether relationships between neurons generate synergistic or redundant information. While it has been long recognized that Shannon information measures on neural activity suffer from a (mostly upward) limited sampling estimation bias, this issue has largely been ignored in the burgeoning field of PID analysis of neural activity. We used simulations to investigate the limited sampling bias of PID computed from discrete probabilities (suited to describe neural spiking activity). We found that PID suffers from a large bias that is uneven across components, with synergy by far the most biased. Using approximate analytical expansions, we found that the bias of synergy increases quadratically with the number of discrete responses of each neuron, whereas the bias of unique and redundant information increase only linearly or sub-linearly. Based on the understanding of the PID bias properties, we developed simple yet effective procedures that correct for the bias effectively, and that improve greatly the PID estimation with respect to current state-of-the-art procedures. We apply these PID bias correction procedures to datasets of 53117 pairs neurons in auditory cortex, posterior parietal cortex and hippocampus of mice performing cognitive tasks, deriving precise estimates and bounds of how synergy and redundancy vary across these brain regions.

Neuroscience
Measuring Information Transfer in Neural Networks

Xiao Zhang,Xingjian Li,Dejing Dou,Ji Wu

2020-01-01

Abstract: Estimation of the information content in a neural network model can be prohibitive, because of difficulty in finding an optimal codelength of the model. We propose to use a surrogate measure to bypass directly estimating model information. The proposed Information Transfer ($L_{IT}$) is a measure of model information based on prequential coding. $L_{IT}$ is theoretically connected to model information, and is consistently correlated with model information in experiments. We show that $L_{IT}$ can be used as a measure of generalizable knowledge in a model or a dataset. Therefore, $L_{IT}$ can serve as an analytical tool in deep learning. We apply $L_{IT}$ to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. $L_{IT}$ provides an informational perspective which helps us discover new insights into neural network learning.
Applications of Information Theory to Analysis of Neural Data

Simon R. Schultz,Robin A. A. Ince,Stefano Panzeri

DOI: https://doi.org/10.1007/978-1-4614-7320-6_280-1

2015-01-08

Abstract:Information theory is a practical and theoretical framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. It has a number of useful properties: it is a general measure sensitive to any relationship, not only linear effects; it has meaningful units which in many cases allow direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single trials, rather than in averages over multiple trials. A variety of information theoretic quantities are commonly used in neuroscience - (see entry "Definitions of Information-Theoretic Quantities"). In this entry we review some applications of information theory in neuroscience to study encoding of information in both single neurons and neuronal populations.

Neurons and Cognition

Methods for Estimating Neural Information

Mutual information analysis on non-stationary neuron importance for brain machine interfaces.

Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization

Neural Joint Entropy Estimation

Correlation in Neuronal Calcium Spiking: Quantification based on Empirical Mutual Information Rate

Rate of mutual information between coarse-grained non-Markovian variables

Determination of relevant neuron-neuron connections for neural prosthetics using time-delayed mutual information: tutorial and preliminary results

Correlated Firing and the Information Represented by Neurons in Short Epochs

Quantifying information stored in synaptic connections rather than in firing patterns of neural networks

Estimating quantum mutual information through a quantum neural network

Summary of Information Theoretic Quantities

Estimating mutual information for spike trains: a bird song example

Correlations and the Encoding of Information in the Nervous System

Does a larger neural network mean greater information transmission efficiency?

An Information Theoretic Approach to the Contributions of the Firing Rates and the Correlations Between the Firing of Neurons.

Quantifying evoked responses through information-theoretical measures

Sampling bias corrections for accurate neural measures of redundant, unique, and synergistic information

Measuring Information Transfer in Neural Networks

Applications of Information Theory to Analysis of Neural Data