The Asymptotic Cost of Complexity

Martin W Cripps

2024-08-27

Abstract:We propose a measure of learning efficiency for non-finite state spaces. We characterize the complexity of a learning problem by the metric entropy of its state space. We then describe how learning efficiency is determined by this measure of complexity. This is, then, applied to two models where agents learn high-dimensional states.

Theoretical Economics,Information Theory

What problem does this paper attempt to address?

Complexity Measures and Concept Learning

Andreas D. Pape,Kenneth J. Kurtz,Hiroki Sayama

DOI: https://doi.org/10.1016/j.jmp.2015.01.001

2015-01-24

Abstract:The nature of concept learning is a core question in cognitive science. Theories must account for the relative difficulty of acquiring different concepts by supervised learners. For a canonical set of six category types, two distinct orderings of classification difficulty have been found. One ordering, which we call paradigm-specific, occurs when adult human learners classify objects with easily distinguishable characteristics such as size, shape, and shading. The general order occurs in all other known cases: when adult humans classify objects with characteristics that are not readily distinguished (e.g., brightness, saturation, hue); for children and monkeys; and when categorization difficulty is extrapolated from errors in identification learning. The paradigm-specific order was found to be predictable mathematically by measuring the logical complexity of tasks, i.e., how concisely the solution can be represented by logical rules. However, logical complexity explains only the paradigm-specific order but not the general order. Here we propose a new difficulty measurement, information complexity, that calculates the amount of uncertainty remaining when a subset of the dimensions are specified. This measurement is based on Shannon entropy. We show that, when the metric extracts minimal uncertainties, this new measurement predicts the paradigm-specific order for the canonical six category types, and when the metric extracts average uncertainties, this new measurement predicts the general order. Moreover, for learning category types beyond the canonical six, we find that the minimal-uncertainty formulation correctly predicts the paradigm-specific order as well or better than existing metrics (Boolean complexity and GIST) in most cases.

Information Theory,Machine Learning
Information Erasure Lurking Behind Measures of Complexity

Karoline Wiesner,Mile Gu,Elisabeth Rieper,Vlatko Vedral

2009-01-01

Abstract:Complex systems are found in most branches of science. It is still argued howto best quantify their complexity and to what end. One prominent measure ofcomplexity (the statistical complexity) has an operational meaning in terms ofthe amount of resources needed to forecasting a system's behaviour. Another one(the effective measure complexity, aka excess entropy) is a measure of mutualinformation stored in the system proper. We show that for any given system thetwo measures differ by the amount of information erased during forecasting. Weinterpret the difference as inefficiency of a given model. We find a bound tothe ratio of the two measures defined as information-processing efficiency, inanalogy to the second law of thermodynamics. This new link between twoprominent measures of complexity provides a quantitative criterion for goodmodels of complex systems, namely those with little information erasure.
A Measure of Learning Model Complexity by VC Dimension

Zongben Xu

2002-01-01

Abstract:When developing models there is always a trade-off between model complexity and model fit. In this paper, a measure of learning model complexity based on VC dimension is presented, and some relevant mathematical theory surrounding the derivation and use of this metric is summarized. The measure allows modelers to control the amount of error that is returned from a modeling system and to state upper bounds on the amount of error that the modeling system will return on all future, as yet unseen and uncollected data sets. It is possible for modelers to use the VC theory to determine which type of model more accurately represents a system.
Metric-Entropy Limits on Nonlinear Dynamical System Learning

Yang Pan,Clemens Hutter,Helmut Bölcskei

2024-07-01

Abstract:This paper is concerned with the fundamental limits of nonlinear dynamical system learning from input-output traces. Specifically, we show that recurrent neural networks (RNNs) are capable of learning nonlinear systems that satisfy a Lipschitz property and forget past inputs fast enough in a metric-entropy optimal manner. As the sets of sequence-to-sequence maps realized by the dynamical systems we consider are significantly more massive than function classes generally considered in deep neural network approximation theory, a refined metric-entropy characterization is needed, namely in terms of order, type, and generalized dimension. We compute these quantities for the classes of exponentially-decaying and polynomially-decaying Lipschitz fading-memory systems and show that RNNs can achieve them.

Machine Learning,Information Theory,Dynamical Systems
The Statistical Complexity of Interactive Decision Making

Dylan J. Foster,Sham M. Kakade,Jian Qian,Alexander Rakhlin

2023-07-12

Abstract:A fundamental challenge in interactive learning and decision making, ranging from bandit problems to reinforcement learning, is to provide sample-efficient, adaptive learning algorithms that achieve near-optimal regret. This question is analogous to the classical problem of optimal (supervised) statistical learning, where there are well-known complexity measures (e.g., VC dimension and Rademacher complexity) that govern the statistical complexity of learning. However, characterizing the statistical complexity of interactive learning is substantially more challenging due to the adaptive nature of the problem. The main result of this work provides a complexity measure, the Decision-Estimation Coefficient, that is proven to be both necessary and sufficient for sample-efficient interactive learning. In particular, we provide: 1. a lower bound on the optimal regret for any interactive decision making problem, establishing the Decision-Estimation Coefficient as a fundamental limit. 2. a unified algorithm design principle, Estimation-to-Decisions (E2D), which transforms any algorithm for supervised estimation into an online algorithm for decision making. E2D attains a regret bound that matches our lower bound up to dependence on a notion of estimation performance, thereby achieving optimal sample-efficient learning as characterized by the Decision-Estimation Coefficient. Taken together, these results constitute a theory of learnability for interactive decision making. When applied to reinforcement learning settings, the Decision-Estimation Coefficient recovers essentially all existing hardness results and lower bounds. More broadly, the approach can be viewed as a decision-theoretic analogue of the classical Le Cam theory of statistical estimation; it also unifies a number of existing approaches -- both Bayesian and frequentist.

Machine Learning,Optimization and Control,Statistics Theory
Complexity of Networks

Russell K. Standish

DOI: https://doi.org/10.48550/arXiv.cs/0508075

2005-08-17

Abstract:Network or graph structures are ubiquitous in the study of complex systems. Often, we are interested in complexity trends of these system as it evolves under some dynamic. An example might be looking at the complexity of a food web as species enter an ecosystem via migration or speciation, and leave via extinction. In this paper, a complexity measure of networks is proposed based on the {\em complexity is information content} paradigm. To apply this paradigm to any object, one must fix two things: a representation language, in which strings of symbols from some alphabet describe, or stand for the objects being considered; and a means of determining when two such descriptions refer to the same object. With these two things set, the information content of an object can be computed in principle from the number of equivalent descriptions describing a particular object. I propose a simple representation language for undirected graphs that can be encoded as a bitstring, and equivalence is a topological equivalence. I also present an algorithm for computing the complexity of an arbitrary undirected network.

Information Theory
Three Perspectives on Complexity $-$ Entropy, Compression, Subsymmetry

Nithin Nagaraj,Karthi Balasubramanian

DOI: https://doi.org/10.1140/epjst/e2016-60347-2

2016-10-31

Abstract:There is no single universally accepted definition of "Complexity". There are several perspectives on complexity and what constitutes complex behaviour or complex systems, as opposed to regular, predictable behaviour and simple systems. In this paper, we explore the following perspectives on complexity: "effort-to-describe" (Shannon entropy $H$, Lempel-Ziv complexity $LZ$), "effort-to-compress" ($ETC$ complexity) and "degree-of-order" (Subsymmetry or $SubSym$). While Shannon entropy and $LZ$ are very popular and widely used, $ETC$ is a recently proposed measure for time series. In this paper, we also propose a novel normalized measure $SubSym$ based on the existing idea of counting the number of subsymmetries or palindromes within a sequence. We compare the performance of these complexity measures on the following tasks: a) characterizing complexity of short binary sequences of lengths 4 to 16, b) distinguishing periodic and chaotic time series from 1D logistic map and 2D Hénon map, and c) distinguishing between tonic and irregular spiking patterns generated from the "Adaptive exponential integrate-and-fire" neuron model. Our study reveals that each perspective has its own advantages and uniqueness while also having an overlap with each other.

Data Analysis, Statistics and Probability,Neurons and Cognition
On Statistical Efficiency in Learning

Jie Ding,Enmao Diao,Jiawei Zhou,Vahid Tarokh

DOI: https://doi.org/10.1109/tit.2020.3047620

IF: 2.5

2021-04-01

IEEE Transactions on Information Theory

Abstract:A central issue of many statistical learning problems is to select an appropriate model from a set of candidate models. Large models tend to inflate the variance (or overfitting), while small models tend to cause biases (or underfitting) for a given fixed dataset. In this work, we address the critical challenge of model selection to strike a balance between model fitting and model complexity, thus gaining reliable predictive power. We consider the task of approaching the theoretical limit of statistical learning, meaning that the selected model has the predictive performance that is as good as the best possible model given a class of potentially misspecified candidate models. We propose a generalized notion of Takeuchi’s information criterion and prove that the proposed method can asymptotically achieve the optimal out-sample prediction loss under reasonable assumptions. It is the first proof of the asymptotic property of Takeuchi’s information criterion to our best knowledge. Our proof applies to a wide variety of nonlinear models, loss functions, and high dimensionality (in the sense that the models’ complexity can grow with sample size). The proposed method can be used as a computationally efficient surrogate for leave-one-out cross-validation. Moreover, for modeling streaming data, we propose an online algorithm that sequentially expands the model complexity to enhance selection stability and reduce computation cost. Experimental studies show that the proposed method has desirable predictive power and significantly less computational cost than some popular methods.

computer science, information systems,engineering, electrical & electronic
Efficiency Bound of Learning with Coarse Graining

Minghao Li,Shihao Xia,Youlin Wang,Minglong Lv,Jincan Chen,Shanhe Su

DOI: https://doi.org/10.1088/0256-307x/40/11/110501

2023-01-01

Chinese Physics Letters

Abstract:A thermodynamic formalism describing the efficiency of information learning is proposed, which is applicable to stochastic thermodynamic systems with multiple internal degrees of freedom. The learning rate, entropy production rate and entropy flow from the system to the environment under coarse-grained dynamics are derived. The Cauchy–Schwarz inequality is applied to demonstrate the lower bound on the entropy production rate of an internal state. The inequality of the entropy production rate is tighter than the Clausius inequality, leading to a derivative of the upper bound on the efficiency of learning. The results are verified in cellular networks with information processes.
Asymptotic Optimality in Stochastic Optimization

John Duchi,Feng Ruan

DOI: https://doi.org/10.48550/arXiv.1612.05612

2018-11-02

Abstract:We study local complexity measures for stochastic convex optimization problems, providing a local minimax theory analogous to that of Hájek and Le Cam for classical statistical problems. We give complementary optimality results, developing fully online methods that adaptively achieve optimal convergence guarantees. Our results provide function-specific lower bounds and convergence results that make precise a correspondence between statistical difficulty and the geometric notion of tilt-stability from optimization. As part of this development, we show how variants of Nesterov's dual averaging---a stochastic gradient-based procedure---guarantee finite time identification of constraints in optimization problems, while stochastic gradient procedures fail. Additionally, we highlight a gap between problems with linear and nonlinear constraints: standard stochastic-gradient-based procedures are suboptimal even for the simplest nonlinear constraints, necessitating the development of asymptotically optimal Riemannian stochastic gradient methods.

Statistics Theory,Optimization and Control,Machine Learning
The Asymptotic Behaviour of Information Leakage Metrics

Sophie Taylor,Praneeth Kumar Vippathalla,Justin P. Coon

2024-09-20

Abstract:Information theoretic leakage metrics quantify the amount of information about a private random variable $X$ that is leaked through a correlated revealed variable $Y$. They can be used to evaluate the privacy of a system in which an adversary, from whom we want to keep $X$ private, is given access to $Y$. Global information theoretic leakage metrics quantify the overall amount of information leaked upon observing $Y$, whilst their pointwise counterparts define leakage as a function of the particular realisation $y$ that the adversary sees, and thus can be viewed as random variables. We consider an adversary who observes a large number of independent identically distributed realisations of $Y$. We formalise the essential asymptotic behaviour of an information theoretic leakage metric, considering in turn what this means for pointwise and global metrics. With the resulting requirements in mind, we take an axiomatic approach to defining a set of pointwise leakage metrics, as well as a set of global leakage metrics that are constructed from them. The global set encompasses many known measures including mutual information, Sibson mutual information, Arimoto mutual information, maximal leakage, min entropy leakage, $f$-divergence metrics, and g-leakage. We prove that both sets follow the desired asymptotic behaviour. Finally, we derive composition theorems which quantify the rate of privacy degradation as an adversary is given access to a large number of independent observations of $Y$. It is found that, for both pointwise and global metrics, privacy degrades exponentially with increasing observations for the adversary, at a rate governed by the minimum Chernoff information between distinct conditional channel distributions. This extends the work of Wu et al. (2024), who have previously found this to be true for certain known metrics, including some that fall into our more general set.

Information Theory
Theoretical bound of the efficiency of learning with coarse-graining

Minghao Li,Shihao Xia,Youlin Wang,Minglong Lv,Shanhe Su

2023-05-30

Abstract:A thermodynamic formalism describing the efficiency of information learning is proposed, which is applicable for stochastic thermodynamic systems with multiple internal degree of freedom. The learning rate, entropy production rate (EPR), and entropy flow from the system to the environment under coarse-grained dynamics are derived. The Cauchy-Schwarz inequality has been applied to demonstrate the lower bound on the EPR of an internal state. The inequality of EPR is tighter than the Clausius inequality, leading to the derivative of the upper bound on the efficiency of learning. The results are verified in cellular networks with information processes.

Statistical Mechanics,Biological Physics
State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?

Ingvar Ziemann,Nikolai Matni,George J. Pappas

2024-09-20

Abstract:How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems -- a simple instance of self-supervised learning -- exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters -- a phenomenon akin to emergence. We also investigate the role of the learner's parametrization and consider a simple version of a linear dynamical system with hidden state -- an imperfectly observed random walk in $\mathbb{R}$. For this situation, we show that there exists no learner using a linear filter which can succesfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.

Machine Learning,Systems and Control
Efficiency of the hidden fermion determinant states Ansatz in the light of different complexity measures

Björn J. Wurst,Dante M. Kennes,Jonas B. Profe

2024-11-07

Abstract:Finding reliable approximations to the quantum many-body problem is one of the central challenges of modern physics. Elemental to this endeavor is the development of advanced numerical techniques pushing the limits of what is tractable. One such recently proposed numerical technique are neural quantum states. This new type of wavefunction based Ansätze utilizes the expressivity of neural networks to tackle fundamentally challenging problems, such as the Mott transition. In this paper we aim to gauge the universalness of one representative of neural network Ansätze, the hidden-fermion slater determinant approach. To this end, we study five different fermionic models each displaying volume law scaling of the entanglement entropy. For these, we correlate the effectiveness of the Ansatz with different complexity measures. Each measure indicates a different complexity in the absence of which a conventional Ansatz becomes efficient. We provide evidence that whenever one of the measures indicates proximity to a parameter region in which a conventional approach would work reliable, the neural network approach also works reliable and efficient. This highlights the great potential, but also challenges for neural network approaches: Finding suitable points in theory space around which to construct the Ansatz in order to be able to efficiently treat models unsuitable for their current designs.

Quantum Physics,Disordered Systems and Neural Networks
Dynamical complexity of short and noisy time series

Nithin Nagaraj,Karthi Balasubramanian

DOI: https://doi.org/10.1140/epjst/e2016-60397-x

2017-01-31

The European Physical Journal Special Topics

Abstract:AbstractShannon entropy has been extensively used for characterizing complexity of time series arising from chaotic dynamical systems and stochastic processes such as Markov chains. However, for short and noisy time series, Shannon entropy performs poorly. Complexity measures which are based on lossless compression algorithms are a good substitute in such scenarios. We evaluate the performance of two such Compression-Complexity Measures namely Lempel-Ziv complexity (LZ) and Effort-To-Compress (ETC) on short time series from chaotic dynamical systems in the presence of noise. Both LZ and ETC outperform Shannon entropy (H) in accurately characterizing the dynamical complexity of such systems. For very short binary sequences (which arise in neuroscience applications), ETC has higher number of distinct complexity values than LZ and H, thus enabling a finer resolution. For two-state ergodic Markov chains, we empirically show that ETC converges to a steady state value faster than LZ. Compression-Complexity measures are promising for applications which involve short and noisy time series.
Assessing complexity by means of maximum entropy models

Gregor Chliamovitch,Bastien Chopard,Lino Velasquez

DOI: https://doi.org/10.48550/arXiv.1408.0368

2014-08-02

Abstract:We discuss a characterization of complexity based on successive approximations of the probability density describing a system by means of maximum entropy methods, thereby quantifying the respective role played by different orders of interaction. This characterization is applied on simple cellular automata in order to put it in perspective with the usual notion of complexity for such systems based on Wolfram classes. The overlap is shown to be good, but not perfect. This suggests that complexity in the sense of Wolfram emerges as an intermediate regime of maximum entropy-based complexity, but also gives insights regarding the role of initial conditions in complexity-related issues.

Cellular Automata and Lattice Gases,Statistical Mechanics,Computational Physics
State Complexity Approximation

Yuan Gao,Sheng Yu

DOI: https://doi.org/10.4204/eptcs.3.11

2009-01-01

Abstract:In this paper, we introduce the new concept of state complexity approximation, which is a further development of state complexity estimation. We show that this new concept is useful in both of the following two cases: the exact state complexities are not known and the state complexities have been obtained but are in incomprehensible form.
Unlearnable Games and "Satisficing" Decisions: A Simple Model for a Complex World

Jérôme Garnier-Brun,Michael Benzaquen,Jean-Philippe Bouchaud

DOI: https://doi.org/10.1103/physrevx.14.021039

2024-06-07

Physical Review X

Abstract:As a schematic model of the complexity economic agents are confronted with, we introduce the "Sherrington-Kirkpatrick game," a discrete time binary choice model inspired from mean-field spin glasses. We show that, even in a completely static environment, agents are unable to learn collectively optimal strategies. This is either because the learning process gets trapped in a suboptimal fixed point or because learning never converges and leads to a never-ending evolution of agent intentions. Contrarily to the hope that learning might save the standard "rational expectation" framework in economics, we argue that complex situations are generically unlearnable and agents must do with satisficing solutions, as argued long ago by Simon [ Q. J. Econ. 69 , 99 (1955)]. Only a centralized, omniscient agent endowed with enormous computing power could qualify to determine the optimal strategy of all agents. Using a mix of analytical arguments and numerical simulations, we find that (i) long memory of past rewards is beneficial to learning, whereas overreaction to recent past is detrimental and leads to cycles or chaos; (ii) increased competition (nonreciprocity) destabilizes fixed points and leads first to chaos and, in the high competition limit, to quasicycles; (iii) some amount of randomness in the learning process, perhaps paradoxically, allows the system to reach better collective decisions; (iv) nonstationary, "aging" behavior spontaneously emerges in a large swath of parameter space of our complex but static world. On the positive side, we find that the learning process allows cooperative systems to coordinate around satisficing solutions with rather high (but markedly suboptimal) average reward. However, hypersensitivity to the game parameters makes it impossible to predict ex ante who will be better or worse off in our stylized economy. The statistical description of the space of satisficing solutions is an open problem. https://doi.org/10.1103/PhysRevX.14.021039 Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. Published by the American Physical Society

physics, multidisciplinary
On the Complexity of Computing Maximum Entropy for Markovian Models.

Taolue Chen,Tingting Han

DOI: https://doi.org/10.4230/lipics.fsttcs.2014.571

2014-01-01

Abstract:We investigate the complexity of computing entropy of various Markovian models including Markov Chains (MCs), Interval Markov Chains (IMCs) and Markov Decision Processes (MDPs). We consider both entropy and entropy rate for general MCs, and study two algorithmic questions, i.e., entropy approximation problem and entropy threshold problem. The former asks for an approximation of the entropy/entropy rate within a given precision, whereas the latter aims to decide whether they exceed a given threshold. We give polynomial-time algorithms for the approximation problem, and show the threshold problem is in P CH3 (hence in PSPACE) and in P assuming some number-theoretic conjectures. Furthermore, we study both questions for IMCs and MDPs where we aim to maximise the entropy/entropy rate among an infinite family of MCs associated with the given model. We give various conditional decidability results for the threshold problem, and show the approximation problem is solvable in polynomial-time via convex programming.
Asymptotic Behavior of Bayesian Learners with Misspecified Models

Ignacio Esponda,Demian Pouzo,Yuichi Yamamoto

DOI: https://doi.org/10.48550/arXiv.1904.08551

2019-10-23

Abstract:We consider an agent who represents uncertainty about the environment via a possibly misspecified model. Each period, the agent takes an action, observes a consequence, and uses Bayes' rule to update her belief about the environment. This framework has become increasingly popular in economics to study behavior driven by incorrect or biased beliefs. Current literature has characterized asymptotic behavior under fairly specific assumptions. By first showing that the key element to predict the agent's behavior is the frequency of her past actions, we are able to characterize asymptotic behavior in general settings in terms of the solutions of a generalization of a differential equation that describes the evolution of the frequency of actions. We then present a series of implications that can be readily applied to economic applications, thus providing off-the-shelf tools that can be used to characterize behavior under misspecified learning.

Theoretical Economics,Statistics Theory

The Asymptotic Cost of Complexity

Complexity Measures and Concept Learning

Information Erasure Lurking Behind Measures of Complexity

A Measure of Learning Model Complexity by VC Dimension

Metric-Entropy Limits on Nonlinear Dynamical System Learning

The Statistical Complexity of Interactive Decision Making

Complexity of Networks

Three Perspectives on Complexity $-$ Entropy, Compression, Subsymmetry

On Statistical Efficiency in Learning

Efficiency Bound of Learning with Coarse Graining

Asymptotic Optimality in Stochastic Optimization

The Asymptotic Behaviour of Information Leakage Metrics

Theoretical bound of the efficiency of learning with coarse-graining

State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?

Efficiency of the hidden fermion determinant states Ansatz in the light of different complexity measures

Dynamical complexity of short and noisy time series

Assessing complexity by means of maximum entropy models

State Complexity Approximation

Unlearnable Games and "Satisficing" Decisions: A Simple Model for a Complex World

On the Complexity of Computing Maximum Entropy for Markovian Models.

Asymptotic Behavior of Bayesian Learners with Misspecified Models