Abstract:Successfully predicting the future states of systems that are complex, stochastic and potentially chaotic is a major challenge. Model forecasting error (FE) is the usual measure of success; however model predictions provide no insights into the potential for improvement. In short, the realized predictability of a specific model is uninformative about whether the system is inherently predictable or whether the chosen model is a poor match for the system and our observations thereof. Ideally, model proficiency would be judged with respect to the systems’ intrinsic predictability – the highest achievable predictability given the degree to which system dynamics are the result of deterministic v. stochastic processes. Intrinsic predictability may be quantified with permutation entropy (PE), a model-free, information-theoretic measure of the complexity of a time series. By means of simulations we show that a correlation exists between estimated PE and FE and show how stochasticity, process error, and chaotic dynamics affect the relationship. This relationship is verified for a dataset of 461 empirical ecological time series. We show how deviations from the expected PE-FE relationship are related to covariates of data quality and the nonlinearity of ecological dynamics. These results demonstrate a theoretically-grounded basis for a model-free evaluation of a system’s intrinsic predictability. Identifying the gap between the intrinsic and realized predictability of time series will enable researchers to understand whether forecasting proficiency is limited by the quality and quantity of their data or the ability of the chosen forecasting model to explain the data. Intrinsic predictability also provides a model-free baseline of forecasting proficiency against which modeling efforts can be evaluated. Glossary Active information: The amount of information that is available to forecasting models (redundant information minus lost information; Fig. 1). Forecasting error (FE): A measure of the discrepancy between a model’s forecasts and the observed dynamics of a system. Common measures of forecast error are root mean squared error and mean absolute error. Entropy: Measures the average amount of information in the outcome of a stochastic process. Information: Any entity that provides answers and resolves uncertainty about a process. When information is calculated using logarithms to the base two (i.e. information in bits), it is the minimum number of yes/no questions required, on average, to determine the identity of the symbol (Jost 2006). The information in an observation consists of information inherited from the past (redundant information), and of new information. Intrinsic predictability: the maximum achievable predictability of a system (Beckage et al. 2011). Lost information: The part of the redundant information lost due to measurement or sampling error, or transformations of the data (Fig. 1). New information, Shannon entropy rate: The Shannon entropy rate quantifies the average amount of information per observation in a time series that is unrelated to the past, i.e., the new information (Fig. 1). Nonlinearity: When the deterministic processes governing system dynamics depend on the state of the system. Permutation entropy (PE): permutation entropy is a measure of the complexity of a time series (Bandt & Pompe, 2002) that is negatively correlated with a system’s predictability (Garland et al. 2015). Permutation entropy quantifies the combined new and lost information. PE is scaled to range between a minimum of 0 and a maximum of 1. Realized predictability: the achieved predictability of a system from a given forecasting model. Redundant information: The information inherited from the past, and thus the maximum amount of information available for use in forecasting (Fig. 1). Symbols, words, permutations: symbols are simply the smallest unit in a formal language such as the letters in the English alphabet i.e., {“A”, “B”,…, “Z”}. In information theory the alphabet is more abstract, such as elements in the set {“up”, “down”} or {“1”, “2”, “3”}. Words, of length m refer to concatenations of the symbols (e.g., up-down-down) in a set. Permutations are the possible orderings of symbols in a set. In this manuscript, the words are the permutations that arise from the numerical ordering of m data points in a time series. Weighted permutation entropy (WPE): a modification of permutation entropy (Fadlallah et al., 2013) that distinguishes between small-scale, noise-driven variation and large-scale, system-driven variation by considering the magnitudes of changes in addition to the rank-order patterns of PE.

A note on a priori forecasting and simplicity bias in time series

Exploring simplicity bias in 1D dynamical systems

Simplicity bias, algorithmic probability, and the random logistic map

Model-free quantification of time-series predictability

Predictability limit of partially observed systems

Enhanced Time Series Predictability with Well-Defined Structures

Detecting stochasticity in population time series using a non‐parametric test of intrinsic predictability

A simple statistical approach to prediction in open high dimensional chaotic systems

Deep Probabilistic Koopman: Long-term time-series forecasting under periodic uncertainties

Dynamic dependence networks: Financial time series forecasting and portfolio decisions (with discussion)

Globally Optimal And Adaptive Short-Term Forecast of Locally Stationary Time Series And A Test for Its Stability

Revisiting the Efficacy of Signal Decomposition in AI-based Time Series Prediction

The Complexity of Sequential Prediction in Dynamical Systems

Theory and Algorithms for Forecasting Time Series

A Survey of Deep Learning and Foundation Models for Time Series Forecasting

Empirical Risk Minimization for Time Series: Nonparametric Performance Bounds for Prediction

The Intrinsic Predictability of Ecological Time Series and Its Potential to Guide Forecasting

Nonparametric forecasting of multivariate probability density functions

Probability of constructing prediction model for observable of a dynamical process via time series

Event prediction and causality inference despite incomplete information