Abstract:Successfully predicting the future states of systems that are complex, stochastic and potentially chaotic is a major challenge. Model forecasting error (FE) is the usual measure of success; however model predictions provide no insights into the potential for improvement. In short, the realized predictability of a specific model is uninformative about whether the system is inherently predictable or whether the chosen model is a poor match for the system and our observations thereof. Ideally, model proficiency would be judged with respect to the systems’ intrinsic predictability – the highest achievable predictability given the degree to which system dynamics are the result of deterministic v. stochastic processes. Intrinsic predictability may be quantified with permutation entropy (PE), a model-free, information-theoretic measure of the complexity of a time series. By means of simulations we show that a correlation exists between estimated PE and FE and show how stochasticity, process error, and chaotic dynamics affect the relationship. This relationship is verified for a dataset of 461 empirical ecological time series. We show how deviations from the expected PE-FE relationship are related to covariates of data quality and the nonlinearity of ecological dynamics. These results demonstrate a theoretically-grounded basis for a model-free evaluation of a system’s intrinsic predictability. Identifying the gap between the intrinsic and realized predictability of time series will enable researchers to understand whether forecasting proficiency is limited by the quality and quantity of their data or the ability of the chosen forecasting model to explain the data. Intrinsic predictability also provides a model-free baseline of forecasting proficiency against which modeling efforts can be evaluated. Glossary Active information: The amount of information that is available to forecasting models (redundant information minus lost information; Fig. 1). Forecasting error (FE): A measure of the discrepancy between a model’s forecasts and the observed dynamics of a system. Common measures of forecast error are root mean squared error and mean absolute error. Entropy: Measures the average amount of information in the outcome of a stochastic process. Information: Any entity that provides answers and resolves uncertainty about a process. When information is calculated using logarithms to the base two (i.e. information in bits), it is the minimum number of yes/no questions required, on average, to determine the identity of the symbol (Jost 2006). The information in an observation consists of information inherited from the past (redundant information), and of new information. Intrinsic predictability: the maximum achievable predictability of a system (Beckage et al. 2011). Lost information: The part of the redundant information lost due to measurement or sampling error, or transformations of the data (Fig. 1). New information, Shannon entropy rate: The Shannon entropy rate quantifies the average amount of information per observation in a time series that is unrelated to the past, i.e., the new information (Fig. 1). Nonlinearity: When the deterministic processes governing system dynamics depend on the state of the system. Permutation entropy (PE): permutation entropy is a measure of the complexity of a time series (Bandt & Pompe, 2002) that is negatively correlated with a system’s predictability (Garland et al. 2015). Permutation entropy quantifies the combined new and lost information. PE is scaled to range between a minimum of 0 and a maximum of 1. Realized predictability: the achieved predictability of a system from a given forecasting model. Redundant information: The information inherited from the past, and thus the maximum amount of information available for use in forecasting (Fig. 1). Symbols, words, permutations: symbols are simply the smallest unit in a formal language such as the letters in the English alphabet i.e., {“A”, “B”,…, “Z”}. In information theory the alphabet is more abstract, such as elements in the set {“up”, “down”} or {“1”, “2”, “3”}. Words, of length m refer to concatenations of the symbols (e.g., up-down-down) in a set. Permutations are the possible orderings of symbols in a set. In this manuscript, the words are the permutations that arise from the numerical ordering of m data points in a time series. Weighted permutation entropy (WPE): a modification of permutation entropy (Fadlallah et al., 2013) that distinguishes between small-scale, noise-driven variation and large-scale, system-driven variation by considering the magnitudes of changes in addition to the rank-order patterns of PE.

Predictability Analysis of Regression Problems via Conditional Entropy Estimations

Expectile regression for analyzing heteroscedasticity in high dimension

Predictability of Equity Returns over Different Time Horizons: A Nonparametric Approach

Model-agnostic variable importance for predictive uncertainty: an entropy-based approach

Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data

Beyond the Norms: Detecting Prediction Errors in Regression Models

Hidden Variable Discovery Based on Regression and Entropy

A review of predictive uncertainty estimation with machine learning

Prediction of Extremal Expectile Based on Regression Models With Heteroscedastic Extremes

Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty

Least Absolute Relative Error Estimation

Model Agnostic Explainable Selective Regression via Uncertainty Estimation

Model-free quantification of time-series predictability

Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors

On Information-Theoretic Measures of Predictive Uncertainty

Variable Selection with Copula Entropy

The Intrinsic Predictability of Ecological Time Series and Its Potential to Guide Forecasting

A Differential Entropy Estimator for Training Neural Networks

Estimating Predictability: Redundancy and Surrogate Data Method

A better measure of relative prediction accuracy for model selection and model estimation