Information Theoretic I-MMSE generalize Time-Frequency Signal Processing Tools

Samah A. M. Ghanem

2023-01-06

Abstract:In this paper, we capitalize on information theoretic-estimation theoretic result, called the I-MMSE [1]-[2] to show that such tool generalizes time-frequency signal processing tools urgent for the analysis of non-stationary non-Gaussian signals.

Signal Processing,Information Theory

What problem does this paper attempt to address?

Extensions of the I-MMSE Relation

Guangyue Han,Jian Song

DOI: https://doi.org/10.48550/arXiv.1506.07230

2015-06-24

Information Theory

Abstract:Unveiling a fundamental link between information theory and estimation theory, the I-MMSE relation by Guo, Shamai and Verdu~\cite{gu05}, together with its numerous extensions, has great theoretical significance and various practical applications. On the other hand, its influences to date have been restricted to channels without feedback or memory, due to the absence of its extensions to such channels. In this paper, we propose extensions of the I-MMSE relation to discrete-time and continuous-time Gaussian channels with feedback and/or memory. Our approach is based on a very simple observation, which can be applied to other scenarios, such as a simple and direct proof of the classical de Bruijn's identity.
An Information-Theoretic Analysis of Discrete-Time Control and Filtering Limitations by the I-MMSE Relationships

Neng Wan,Dapeng Li,Naira Hovakimyan,Petros G. Voulgaris

2023-04-19

Abstract:Fundamental limitations or performance trade-offs/limits are important properties and constraints of control and filtering systems. Among various trade-off metrics, total information rate, which characterizes the sensitivity trade-offs and average performance of control and filtering systems, is conventionally studied by using the (differential) entropy rate and Kolmogorov-Bode formula. In this paper, by extending the famous I-MMSE (mutual information -- minimum mean-square error) relationship to the discrete-time additive white Gaussian channels with and without feedback, a new paradigm is introduced to estimate and analyze total information rate as a control and filtering trade-off metric. Under this framework, we enrich the trade-off properties of total information rate for a variety of discrete-time control and filtering systems, e.g., LTI, LTV, and nonlinear, and also provide an alternative approach to investigate total information rate via optimal estimation.

Systems and Control,Information Theory,Optimization and Control
Understanding the Method of Interval Errors from the Information Theory Perspective

De Meng,Wen Xu,Menglu Xia

DOI: https://doi.org/10.1109/icassp.2009.4960034

2009-01-01

ICASSP

Abstract:Nonlinear parameter estimation often displays a threshold phenomenon, that is, below certain signal-to-noise ratio (SNR) the estimation mean-square error (MSE) increases dramatically. The method of interval errors (MIE) has been shown to provide accurate MSE prediction of related nonlinear techniques well into the estimation threshold region, yet relatively simple and robust in evaluation compared to a global performance bound. However those features have not been understood on a strict theoretical basis. This paper investigates numerical sensitivity of the MIE to parameter sampling resolution, aiming to understanding, from information theory perspective, the underlying mechanism leading to robust MSE approximation. A recently-developed information theory resolution bound is reinterpreted and applied to specify the parameter sampling resolution. Numerical evaluation of the relevant results for array-based bearing estimation supports the proposed connection between the resolution bound and the MIE.
Information theoretic results for stationary time series and the Gaussian-generalized von Mises time series

Riccardo Gatto

DOI: https://doi.org/10.48550/arXiv.2101.08529

2021-01-21

Abstract:This chapter presents some novel information theoretic results for the analysis of stationary time series in the frequency domain. In particular, the spectral distribution that corresponds to the most uncertain or unpredictable time series with some values of the autocovariance function fixed, is the generalized von Mises spectral distribution. It is thus a maximum entropy spectral distribution and the corresponding stationary time series is called the generalized von Mises time series. The generalized von Mises distribution is used in directional statistics for modelling planar directions that follow a multimodal distribution. Furthermore, the Gaussian-generalized von Mises times series is presented as the stationary time series that maximizes entropies in frequency and time domains, respectively referred to as spectral and temporal entropies. Parameter estimation and some computational aspects with this time series are briefly analyzed.

Statistics Theory
Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

Neri Merhav

DOI: https://doi.org/10.1109/TIT.2011.2132590

2009-11-17

Abstract:In continuation to a recent work on the statistical--mechanical analysis of minimum mean square error (MMSE) estimation in Gaussian noise via its relation to the mutual information (the I-MMSE relation), here we propose a simple and more direct relationship between optimum estimation and certain information measures (e.g., the information density and the Fisher information), which can be viewed as partition functions and hence are amenable to analysis using statistical--mechanical techniques. The proposed approach has several advantages, most notably, its applicability to general sources and channels, as opposed to the I-MMSE relation and its variants which hold only for certain classes of channels (e.g., additive white Gaussian noise channels). We then demonstrate the derivation of the conditional mean estimator and the MMSE in a few examples. Two of these examples turn out to be generalizable to a fairly wide class of sources and channels. For this class, the proposed approach is shown to yield an approximate conditional mean estimator and an MMSE formula that has the flavor of a single-letter expression. We also show how our approach can easily be generalized to situations of mismatched estimation.

Information Theory
The Benefits of Electromagnetic Information Theory for Channel Estimation

Jieao Zhu,Xiaofeng Su,Zhongzhichao Wan,Linglong Dai,Tie Jun Cui

DOI: https://doi.org/10.1109/icc51166.2024.10622245

2024-01-01

Abstract:Electromagnetic information theory (EIT) is an emerging interdisciplinary subject that integrates classical Maxwell electromagnetics and Shannon information theory. The goal of EIT is to uncover the information transmission mechanisms from an electromagnetic (EM) perspective in wireless systems. Existing works on EIT are mainly focused on the analysis of degrees-of-freedom (DoF), system capacity, and characteristics of the electromagnetic channel. However, these works do not clarify whether EIT can improve wireless communication systems. To answer this question, in this paper, we provide a novel example of how to improve channel estimators by integrating EM knowledge into the classical MMSE channel estimator. Specifically, the EM knowledge is first encoded into a spatial correlation function (SCF) of the channel, which we term the EM kernel. This EM kernel plays the role of side information to the channel estimator. Since the EM kernel takes the form of Gaussian processes (GP), we propose the EIT-based Gaussian process regression (EIT-GPR) to derive the channel estimations. Furthermore, we propose EM kernel learning to fit the EM kernel to channel observations. Simulation results show that EIT benefits the channel estimator and enables it to outperform traditional isotropic MMSE algorithm, thus proving the practical values of EIT.
Beyond the I-MMSE relation: derivatives of mutual information in Gaussian channels

Minh-Toan Nguyen

DOI: https://doi.org/10.1109/TIT.2024.3447224

2024-08-25

Abstract:The I-MMSE formula connects two important quantities in information theory and estimation theory: the mutual information and the minimum mean-squared error (MMSE). It states that in a scalar Gaussian channel, the derivative of the mutual information with respect to the signal-to-noise ratio (SNR) is one-half of the MMSE. Although any derivative at a fixed order can be computed in principle, a general formula for all the derivatives is still unknown. In this paper, we derive this general formula for vector Gaussian channels. The obtained result is remarkably similar to the classic cumulant-moment relation in statistical theory.

Information Theory
Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Shengtian Yang,Thomas Honold,Zhaoyang Zhang

DOI: https://doi.org/10.1109/wcsp.2016.7752489

2016-01-01

Abstract:A general approach is established for deriving one-shot performance bounds for information-theoretic problems on general alphabets beyond countable alphabets. It is mainly based on the quantization idea and a novel form of "likelihood ratio". As an example, one-shot lower and upper bounds for random number generation from correlated sources on general alphabets are derived.
Intrinsic Information-Theoretic Models

D. Bernal-Casas,J. M. Oller

DOI: https://doi.org/10.3390/e26050370

IF: 2.738

2024-04-29

Entropy

Abstract:With this follow-up paper, we continue developing a mathematical framework based on information geometry for representing physical objects. The long-term goal is to lay down informational foundations for physics, especially quantum physics. We assume that we can now model information sources as univariate normal probability distributions N(μ,σ0), as before, but with a constant σ0 not necessarily equal to 1. Then, we also relaxed the independence condition when modeling m sources of information. Now, we model m sources with a multivariate normal probability distribution Nm(μ,Σ0) with a constant variance–covariance matrix Σ0 not necessarily diagonal, i.e., with covariance values different to 0, which leads to the concept of modes rather than sources. Invoking Schrödinger's equation, we can still break the information into m quantum harmonic oscillators, one for each mode, and with energy levels independent of the values of σ0, altogether leading to the concept of "intrinsic". Similarly, as in our previous work with the estimator's variance, we found that the expectation of the quadratic Mahalanobis distance to the sample mean equals the energy levels of the quantum harmonic oscillator, being the minimum quadratic Mahalanobis distance at the minimum energy level of the oscillator and reaching the "intrinsic" Cramér–Rao lower bound at the lowest energy level. Also, we demonstrate that the global probability density function of the collective mode of a set of m quantum harmonic oscillators at the lowest energy level still equals the posterior probability distribution calculated using Bayes' theorem from the sources of information for all data values, taking as a prior the Riemannian volume of the informative metric. While these new assumptions certainly add complexity to the mathematical framework, the results proven are invariant under transformations, leading to the concept of "intrinsic" information-theoretic models, which are essential for developing physics.

physics, multidisciplinary
Entropy, Information Theory, Information Geometry and Bayesian Inference in Data, Signal and Image Processing and Inverse Problems

Ali Mohammad-Djafari

DOI: https://doi.org/10.3390/e17063989

IF: 2.738

2015-06-12

Entropy

Abstract:The main content of this review article is first to review the main inference tools using Bayes rule, the maximum entropy principle (MEP), information theory, relative entropy and the Kullback–Leibler (KL) divergence, Fisher information and its corresponding geometries. For each of these tools, the precise context of their use is described. The second part of the paper is focused on the ways these tools have been used in data, signal and image processing and in the inverse problems, which arise in different physical sciences and engineering applications. A few examples of the applications are described: entropy in independent components analysis (ICA) and in blind source separation, Fisher information in data model selection, different maximum entropy-based methods in time series spectral estimation and in linear inverse problems and, finally, the Bayesian inference for general inverse problems. Some original materials concerning the approximate Bayesian computation (ABC) and, in particular, the variational Bayesian approximation (VBA) methods are also presented. VBA is used for proposing an alternative Bayesian computational tool to the classical Markov chain Monte Carlo (MCMC) methods. We will also see that VBA englobes joint maximum a posteriori (MAP), as well as the different expectation-maximization (EM) algorithms as particular cases.

physics, multidisciplinary
Ensemble Improved Permutation Entropy: A New Approach for Time Series Analysis

Zhe Chen,Xiaodong Ma,Jielin Fu,Yaan Li

DOI: https://doi.org/10.3390/e25081175

2023-08-07

Abstract:Entropy quantification approaches have gained considerable attention in engineering applications. However, certain limitations persist, including the strong dependence on parameter selection, limited discriminating power, and low robustness to noise. To alleviate these issues, this paper introduces two novel algorithms for time series analysis: the ensemble improved permutation entropy (EIPE) and multiscale EIPE (MEIPE). Our approaches employ a new symbolization process that considers both permutation relations and amplitude information. Additionally, the ensemble technique is utilized to reduce the dependence on parameter selection. We performed a comprehensive evaluation of the proposed methods using various synthetic and experimental signals. The results illustrate that EIPE is capable of distinguishing white, pink, and brown noise with a smaller number of samples compared to traditional entropy algorithms. Furthermore, EIPE displays the potential to discriminate between regular and non-regular dynamics. Notably, when compared to permutation entropy, weighted permutation entropy, and dispersion entropy, EIPE exhibits superior robustness against noise. In practical applications, such as RR interval data classification, bearing fault diagnosis, marine vessel identification, and electroencephalographic (EEG) signal classification, the proposed methods demonstrate better discriminating power compared to conventional entropy measures. These promising findings validate the effectiveness and potential of the algorithms proposed in this paper.
MIMO Capacity Analysis and Channel Estimation for Electromagnetic Information Theory

Jieao Zhu,Vincent Y. F. Tan,Linglong Dai

2024-06-07

Abstract:Electromagnetic information theory (EIT) is an interdisciplinary subject that serves to integrate deterministic electromagnetic theory with stochastic Shannon's information theory. Existing EIT analysis operates in the continuous space domain, which is not aligned with the practical algorithms working in the discrete space domain. This mismatch leads to a significant difficulty in application of EIT methodologies to practical discrete space systems, which is called as the discrete-continuous gap in this paper. To bridge this gap, we establish the discrete-continuous correspondence with a prolate spheroidal wave function (PSWF)-based ergodic capacity analysis framework. Specifically, we state and prove some discrete-continuous correspondence lemmas to establish a firm theoretical connection between discrete information-theoretic quantities to their continuous counterparts. With these lemmas, we apply the PSWF ergodic capacity bound to advanced MIMO architectures such as continuous-aperture MIMO (CAP-MIMO) and extremely large-scale MIMO (XL-MIMO). From this PSWF capacity bound, we discover the capacity saturation phenomenon both theoretically and empirically. Although the growth of MIMO performance is fundamentally limited in this EIT-based analysis framework, we reveal new opportunities in MIMO channel estimation by exploiting the EIT knowledge about the channel. Inspired by the PSWF capacity bound, we utilize continuous PSWFs to improve the pilot design of discrete MIMO channel estimators, which is called as the PSWF channel estimator (PSWF-CE). Simulation results demonstrate improved performances of the proposed PSWF-CE, compared to traditional minimum mean squared error (MMSE) and compressed sensing-based estimators.

Information Theory,Signal Processing
On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Manuel Álvarez Chaves,Hoshin V Gupta,Uwe Ehret,Anneli Guthke

DOI: https://doi.org/10.3390/e26050387

2024-04-30

Abstract:Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.
Assessment of Multivariate Information Transmission in Space-Time-Frequency Domain: A Case Study for EEG Signals

Sixuan He,Yaru Li,Xinning Le,Xiaoyu Han,Jingfeng Lin,Xiaohang Peng,Min Li,Ruobing Yang,Dezhong Yao,Pedro A. Valdes-Sosa,Peng Ren

DOI: https://doi.org/10.1109/tnsre.2023.3260143

IF: 4.9

2023-04-01

IEEE Transactions on Neural Systems and Rehabilitation Engineering

Abstract:Objective: Multivariate signal (MS) analysis, especially the assessment of its information transmission (for example, from the perspective of network science), is the key to our understanding of various phenomena in biology, physics and economics. Although there is a large amount of literature demonstrating that MS can be decomposed into space-time-frequency domain information, there seems to be no research confirming that multivariate information transmission (MIT) in these three domains can be quantified. Therefore, in this study, we attempted to combine dynamic mode decomposition (DMD) and parallel communication model (PCM) together to realize it. Methods: We first regarded MS as a large-scale system and then used DMD to decompose it into specific subsystems with their own intrinsic oscillatory frequencies. At the same time, the transition probability matrix (TPM) of information transmission within and between MS at two consecutive moments in each subsystem can also be calculated. Then, communication parameters (CPs) derived from each TPM were calculated in order to quantify the MIT in the space-time-frequency domain. In this study, multidimensional electroencephalogram (EEG) signals were used to illustrate our method. Results: Compared with traditional EEG brain networks, this method shows greater potential in EEG analysis to distinguish between patients and healthy controls. Conclusion: This study demonstrates the feasibility of measuring MIT in the space-time-frequency domain simultaneously. Significance: This study shows that MIT analysis in the space-time-frequency domain is not only completely different from the MS decomposition in these three domains, but also can reveal many new phenomena behind MS that have not yet been discovered.

engineering, biomedical,rehabilitation
Quantifying evoked responses through information-theoretical measures

Julian Fuhrer,Kyrre Glette,Anaïs Llorens,Tor Endestad,Anne-Kristin Solbakk,Alejandro Omar Blenkmann

DOI: https://doi.org/10.3389/fninf.2023.1128866

2023-05-23

Abstract:Information theory is a viable candidate to advance our understanding of how the brain processes information generated in the internal or external environment. With its universal applicability, information theory enables the analysis of complex data sets, is free of requirements about the data structure, and can help infer the underlying brain mechanisms. Information-theoretical metrics such as Entropy or Mutual Information have been highly beneficial for analyzing neurophysiological recordings. However, a direct comparison of the performance of these methods with well-established metrics, such as the t-test, is rare. Here, such a comparison is carried out by evaluating the novel method of Encoded Information with Mutual Information, Gaussian Copula Mutual Information, Neural Frequency Tagging, and t-test. We do so by applying each method to event-related potentials and event-related activity in different frequency bands originating from intracranial electroencephalography recordings of humans and marmoset monkeys. Encoded Information is a novel procedure that assesses the similarity of brain responses across experimental conditions by compressing the respective signals. Such an information-based encoding is attractive whenever one is interested in detecting where in the brain condition effects are present.
JIDT: An information-theoretic toolkit for studying the dynamics of complex systems

Joseph T. Lizier

DOI: https://doi.org/10.3389/frobt.2014.00011

2014-12-03

Abstract:Complex systems are increasingly being viewed as distributed information processing systems, particularly in the domains of computational neuroscience, bioinformatics and Artificial Life. This trend has resulted in a strong uptake in the use of (Shannon) information-theoretic measures to analyse the dynamics of complex systems in these fields. We introduce the Java Information Dynamics Toolkit (JIDT): a Google code project which provides a standalone, (GNU GPL v3 licensed) open-source code implementation for empirical estimation of information-theoretic measures from time-series data. While the toolkit provides classic information-theoretic measures (e.g. entropy, mutual information, conditional mutual information), it ultimately focusses on implementing higher-level measures for information dynamics. That is, JIDT focusses on quantifying information storage, transfer and modification, and the dynamics of these operations in space and time. For this purpose, it includes implementations of the transfer entropy and active information storage, their multivariate extensions and local or pointwise variants. JIDT provides implementations for both discrete and continuous-valued data for each measure, including various types of estimator for continuous data (e.g. Gaussian, box-kernel and Kraskov-Stoegbauer-Grassberger) which can be swapped at run-time due to Java's object-oriented polymorphism. Furthermore, while written in Java, the toolkit can be used directly in MATLAB, GNU Octave, Python and other environments. We present the principles behind the code design, and provide several examples to guide users.

Information Theory,Mathematical Software,Social and Information Networks,Adaptation and Self-Organizing Systems,Data Analysis, Statistics and Probability
Ensemble equivalence and asymptotic equipartition property in information theory

Meizhu Li,Qi Zhang

2023-12-19

Abstract:There is a consensus in science that information theory and statistical physics have a close relationship but the literary proofs of the equivalence between most of the conceptions in the two disciplines are still missing. In this work, according to the statistical ensembles' description of the information sequences that are generated by the i.i.d. single variable and multivariate information source, the relationship between the ensemble equivalence and asymptotic equipartition property is established. We find that the description of information sequences in classical information theory is a special case of the canonical ensemble description of information sequences. They are both the maximum entropy approximation of the real signal generation. Vice versa, the conjugate microcanonical ensemble description of the information sequences under hard constraints satisfies the condition in real signal generation exactly. Thus, the microcanoincal ensemble description is closer to the real signal generation than the conjugate canonical ensemble, but the ensemble equivalence between the microcanonical and canonical ensemble in the thermodynamic limit guarantees the effectiveness of classical information theory, i.e., the asymptotic equipartition property in information theory is an isotope of ensemble equivalence from statistical physics.

Statistical Mechanics
Electromagnetic Information Theory: Fundamentals, Modeling, Applications, and Open Problems

Jieao Zhu,Zhongzhichao Wan,Linglong Dai,Merouane Debbah,H. Vincent Poor

DOI: https://doi.org/10.1109/mwc.019.2200602

IF: 12.777

2024-01-01

IEEE Wireless Communications

Abstract:Traditional massive multiple-input multiple- output (MIMO) information theory adopts non-physically consistent assumptions, including white-noised, scalar-quantity, far-field, discretized, and monochromatic EM fields, which mismatch the nature of the underlying electromagnetic (EM) fields supporting the physical layer of wireless communication systems. To incorporate EM laws into designing procedures of the physical layer, we first propose the novel concept of EM physical layer, whose backbone theory is called EM information theory (EIT). In this article, we systematically investigate the basic ideas and main results of EIT. First, we review the fundamental analytical tools of classical information theory and EM theory. Then, we introduce the modeling and analysis methodologies of EIT, including continuous field modeling, degrees of freedom, and mutual information analyses. Several EIT-inspired applications are discussed to illustrate how EIT guides the design of practical wireless systems. Finally, we point out the open problems of EIT, where further research efforts are required for EIT to construct a unified interdisciplinary theory.

computer science, information systems,telecommunications,engineering, electrical & electronic, hardware & architecture
Improved multivariate multiscale sample entropy and its application in multi-channel data

Weijia Li,Xiaohong Shen,Yaan Li,Zhe Chen

DOI: https://doi.org/10.1063/5.0150205

2023-06-01

Abstract:Entropy, as a nonlinear feature in information science, has drawn much attention for time series analysis. Entropy features have been used to measure the complexity behavior of time series. However, traditional entropy methods mainly focus on one-dimensional time series originating from single-channel transducers and are incapable of handling the multidimensional time series from multi-channel transducers. Previously, the multivariate multiscale sample entropy (MMSE) algorithm was introduced for multi-channel data analysis. Although MMSE generalizes multiscale sample entropy and provides a new method for multidimensional data analysis, it lacks necessary theoretical support and has shortcomings, such as missing cross-channel correlation information and having biased estimation results. This paper proposes an improved multivariate multiscale sample entropy (IMMSE) algorithm to overcome these shortcomings. This paper highlights the existing shortcomings in MMSE under the generalized algorithm. The rationality of IMMSE is theoretically proven using probability theory. Simulations and real-world data analysis have shown that IMMSE is capable of effectively extracting cross-channel correlation information and demonstrating robustness in practical applications. Moreover, it provides theoretical support for generalizing single-channel entropy methods to multi-channel situations.

mathematics, applied,physics, mathematical

Information Theoretic I-MMSE generalize Time-Frequency Signal Processing Tools

Extensions of the I-MMSE Relation

An Information-Theoretic Analysis of Discrete-Time Control and Filtering Limitations by the I-MMSE Relationships

Understanding the Method of Interval Errors from the Information Theory Perspective

Information theoretic results for stationary time series and the Gaussian-generalized von Mises time series

Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

The Benefits of Electromagnetic Information Theory for Channel Estimation

Beyond the I-MMSE relation: derivatives of mutual information in Gaussian channels

Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Intrinsic Information-Theoretic Models

Entropy, Information Theory, Information Geometry and Bayesian Inference in Data, Signal and Image Processing and Inverse Problems

Ensemble Improved Permutation Entropy: A New Approach for Time Series Analysis

MIMO Capacity Analysis and Channel Estimation for Electromagnetic Information Theory

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data

Assessment of Multivariate Information Transmission in Space-Time-Frequency Domain: A Case Study for EEG Signals

Quantifying evoked responses through information-theoretical measures

JIDT: An information-theoretic toolkit for studying the dynamics of complex systems

Ensemble equivalence and asymptotic equipartition property in information theory

Electromagnetic Information Theory: Fundamentals, Modeling, Applications, and Open Problems

Improved multivariate multiscale sample entropy and its application in multi-channel data