Algorithmic Information Theory and Foundations of Probability

Alexander Shen

DOI: https://doi.org/10.48550/arXiv.0906.4411

2009-06-24

History and Overview

Abstract:The use of algorithmic information theory (Kolmogorov complexity theory) to explain the relation between mathematical probability theory and `real world' is discussed.

What problem does this paper attempt to address?

Foundations of Probability Theory for AI - The Application of Algorithmic Probability to Problems in Artificial Intelligence

Ray Solomonoff

DOI: https://doi.org/10.48550/arXiv.1304.3424

2013-03-28

Abstract:This paper covers two topics: first an introduction to Algorithmic Complexity Theory: how it defines probability, some of its characteristic properties and past successful applications. Second, we apply it to problems in A.I. - where it promises to give near optimum search procedures for two very broad classes of problems.

Artificial Intelligence
Algorithmic statistics revisited

Nikolay Vereshchagin,Alexander Shen

DOI: https://doi.org/10.48550/arXiv.1504.04950

2015-04-20

Information Theory

Abstract:The mission of statistics is to provide adequate statistical hypotheses (models) for observed data. But what is an "adequate" model? To answer this question, one needs to use the notions of algorithmic information theory. It turns out that for every data string $x$ one can naturally define "stochasticity profile", a curve that represents a trade-off between complexity of a model and its adequacy. This curve has four different equivalent definitions in terms of (1)~randomness deficiency, (2)~minimal description length, (3)~position in the lists of simple strings and (4)~Kolmogorov complexity with decompression time bounded by busy beaver function. We present a survey of the corresponding definitions and results relating them to each other.
Foundations of abstract probability theory

Yurii Yurchenko

DOI: https://doi.org/10.48550/arXiv.2110.02009

2021-10-05

Probability

Abstract:Using the ideas of abstract algebra, we introduce the basic concepts of abstract probability theory that generalize the Kolmogorov's probability theory, possibility theory and other theories that deal with uncertainty. Based on abstract addition and multiplication, we define an abstract measure and abstract Lebesgue integral. System of Kolmogorov's axioms is criticized, after which we introduce an abstract probability measure and abstract conditional probability, show that they have recognizable probability properties. In addition, we define an abstract expected value operator as the abstract Lebesgue integral and prove its properties.
Probabilistic foundations of quantum mechanics and quantum information

Andrei Khrennikov

DOI: https://doi.org/10.48550/arXiv.quant-ph/0309066

2003-09-08

Quantum Physics

Abstract:We discuss foundation of quantum mechanics (interpretations, superposition, principle of complementarity, locality, hidden variables) and quantum information theory.
On the Algorithmic Information Between Probabilities

Samuel Epstein

2024-09-12

Abstract:We extend algorithmic conservation inequalities to probability measures. The amount of self information of a probability measure cannot increase when submitted to randomized processing. This includes (potentially non-computable) measures over finite sequences, infinite sequences, and $T_0$, second countable topologies. One example is the convolution of signals over real numbers with probability kernels. Thus the smoothing of any signal due We show that given a quantum measurement, for an overwhelming majority of pure states, no meaningful information is produced.

Computational Complexity,Quantum Physics
Diverse Consequences of Algorithmic Probability

Eray Özkural

DOI: https://doi.org/10.48550/arXiv.1107.2788

2011-11-08

Abstract:We reminisce and discuss applications of algorithmic probability to a wide range of problems in artificial intelligence, philosophy and technological society. We propose that Solomonoff has effectively axiomatized the field of artificial intelligence, therefore establishing it as a rigorous scientific discipline. We also relate to our own work in incremental machine learning and philosophy of complexity.

Information Theory,Artificial Intelligence,Computers and Society
Foundations of non-commutative probability theory (Extended abstract)

Daniel Lehmann

DOI: https://doi.org/10.1145/1562814.1562841

2023-06-02

Abstract:Kolmogorov's setting for probability theory is given an original generalization to account for probabilities arising from Quantum Mechanics. The sample space has a central role in this presentation and random variables, i.e., observables, are defined in a natural way.The mystery presented by the algebraic equations satisfied by (non-commuting) observables that cannot be observed in the same states is elucidated.

Quantum Physics,Probability
Introduction to foundations of probability and randomness (for students in physics), Lectures given at the Institute of Quantum Optics and Quantum Information, Austrian Academy of Science, Lecture-1: Kolmogorov and von Mises

Andrei Khrennikov

DOI: https://doi.org/10.48550/arXiv.1410.5773

2014-10-19

Abstract:The education system for students in physics suffers (worldwide) from the absence of a deep course in probability and randomness. This is the real problem for students interested in quantum information theory, quantum optics, and quantum foundations. Here the primitive treatment of probability and randomness may lead to deep misunderstandings of theory and wrong interpretations of experimental results. Since during my visits (in 2013 and 2014) to IQOQI a number of students (experimenters!) asked me permanently about foundational problems of probability and randomness, especially inter-relation between classical and quantum structures, this year I gave two lectures on these problems. Surprisingly the interest of experiment-oriented students to mathematical peculiarities was very high. This (as well as permanent reminding of prof. Zeilinger) motivated me to write a text based on these lectures which were originally presented in the traditional black-board form. I hope that this might be useful for students from IQOQI as well as other young physicists.

Quantum Physics,Physics Education
Methods of Information Theory and Algorithmic Complexity for Network Biology

Hector Zenil,Narsis A. Kiani,Jesper Tegnér

DOI: https://doi.org/10.48550/arXiv.1401.3604

2015-12-11

Abstract:We survey and introduce concepts and tools located at the intersection of information theory and network biology. We show that Shannon's information entropy, compressibility and algorithmic complexity quantify different local and global aspects of synthetic and biological data. We show examples such as the emergence of giant components in Erdos-Renyi random graphs, and the recovery of topological properties from numerical kinetic properties simulating gene expression data. We provide exact theoretical calculations, numerical approximations and error estimations of entropy, algorithmic probability and Kolmogorov complexity for different types of graphs, characterizing their variant and invariant properties. We introduce formal definitions of complexity for both labeled and unlabeled graphs and prove that the Kolmogorov complexity of a labeled graph is a good approximation of its unlabeled Kolmogorov complexity and thus a robust definition of graph complexity.

Molecular Networks,Quantitative Methods
Information, complexity, brains and reality (Kolmogorov Manifesto)

Giulio Ruffini

DOI: https://doi.org/10.48550/arXiv.0704.1147

2007-04-09

General Physics

Abstract:I discuss several aspects of information theory and its relationship to physics and neuroscience. The unifying thread of this somewhat chaotic essay is the concept of Kolmogorov or algorithmic complexity (Kolmogorov Complexity, for short). I argue that it is natural to interpret cognition as the art of finding algorithms that apprach the Solomonoff-Kolmogorov-Chaitin (algorithmic) Complexity limit with appropriate tradeoffs. In addition, I claim that what we call the universe is an interpreted abstraction--a mental construct--based on the observed coherence between multiple sensory input streams and our own interactions. Hence, the notion of Universe is itself a model.
Kolmogorov complexity in perspective

Marie Ferbus-Zanda,Serge Grigorieff

DOI: https://doi.org/10.48550/arXiv.0801.0354

2008-01-02

Abstract:We survey the diverse approaches to the notion of information content: from Shannon entropy to Kolmogorov complexity. The main applications of Kolmogorov complexity are presented namely, the mathematical notion of randomness (which goes back to the 60's with the work of Martin-Lof, Schnorr, Chaitin, Levin), and classification, which is a recent idea with provocative implementation by Vitanyi and Cilibrasi.

Logic,Computational Complexity,Information Theory
Epistemology as Information Theory: From Leibniz to Omega

G. J. Chaitin

DOI: https://doi.org/10.48550/arXiv.math/0506552

2005-06-28

Abstract:In 1686 in his Discours de Metaphysique, Leibniz points out that if an arbitrarily complex theory is permitted then the notion of "theory" becomes vacuous because there is always a theory. This idea is developed in the modern theory of algorithmic information, which deals with the size of computer programs and provides a new view of Godel's work on incompleteness and Turing's work on uncomputability. Of particular interest is the halting probability Omega, whose bits are irreducible, i.e., maximally unknowable mathematical facts. More generally, these ideas constitute a kind of "digital philosophy" related to recent attempts of Edward Fredkin, Stephen Wolfram and others to view the world as a giant computer. There are also connections with recent "digital physics" speculations that the universe might actually be discrete, not continuous. This systeme du monde is presented as a coherent whole in my book Meta Math!, which will be published this fall.

History and Overview
Quantum Algorithmic Entropy

Peter Gacs

DOI: https://doi.org/10.1088/0305-4470/34/35/312

2001-06-16

Abstract:We extend algorithmic information theory to quantum mechanics, taking a universal semicomputable density matrix (``universal probability'') as a starting point, and define complexity (an operator) as its negative logarithm. A number of properties of Kolmogorov complexity extend naturally to the new domain. Approximately, a quantum state is simple if it is within a small distance from a low-dimensional subspace of low Kolmogorov complexity. The von Neumann entropy of a computable density matrix is within an additive constant from the average complexity. Some of the theory of randomness translates to the new domain. We explore the relations of the new quantity to the quantum Kolmogorov complexity defined by Vitanyi (we show that the latter is sometimes as large as 2n - 2log n and the qubit complexity defined by Berthiaume, Dam and Laplante. The ``cloning'' properties of our complexity measure are similar to those of qubit complexity.

Quantum Physics
Is Randomness "Native" to Computer Science?

Marie Ferbus-Zanda,Serge Grigorieff

DOI: https://doi.org/10.48550/arXiv.0801.0289

2008-01-01

Abstract:We survey the Kolmogorov's approach to the notion of randomness through the Kolmogorov complexity theory. The original motivation of Kolmogorov was to give up a quantitative definition of information. In this theory, an object is randomness in the sense that it has a large information content. Afterwards, we present parts of the work of Martin-Lof, Schnorr, Chaitin and Levin which supply a mathematical notion of randomness throughout diverse theories from the the 60' up to recently.

Logic,Computational Complexity
Information vs Dimension -- an Algorithmic Perspective

Jan Reimann

DOI: https://doi.org/10.1142/9789813228238_0004

2024-08-09

Abstract:This paper surveys work on the relation between fractal dimensions and algorithmic information theory over the past thirty years. It covers the basic development of prefix-free Kolmogorov complexity from an information theoretic point of view, before introducing Hausdorff measures and dimension along with some important examples. The main goal of the paper is to motivate and develop the informal identity "entropy = complexity = dimension" from first principles. The last section of the paper presents some new observations on multifractal measures from an algorithmic viewpoint.

Logic
Kolmogorov complexity as a hidden factor of scientific discourse: from Newton's law to data mining

Yuri I. Manin

DOI: https://doi.org/10.48550/arXiv.1301.0081

2013-01-01

Abstract:The word "complexity" is most often used as a meta--linguistic expression referring to certain intuitive characteristics of a natural system and/or its scientific description. These characteristics may include: sheer amount of data that must be taken into account; visible "chaotic" character of these data and/or space distribution/time evolution of a system etc. This talk is centered around the precise mathematical notion of "Kolmogorov complexity", originated in the early theoretical computer science and measuring the degree to which an available information can be compressed. In the first part, I will argue that a characteristic feature of basic scientific theories, from Ptolemy's epicycles to the Standard Model of elementary particles, is their splitting into two very distinct parts: the part of relatively small Kolmogorov complexity ("laws", "basic equations", "periodic table", "natural selection, genotypes, mutations") and another part, of indefinitely large Kolmogorov complexity ("initial and boundary conditions", "phenotypes", "populations"). The data constituting this latter part are obtained by planned observations, focussed experiments, and afterwards collected in growing databases (formerly known as "books", "tables", "encyclopaedias" etc). In this discussion Kolomogorov complexity plays a role of the central metaphor. The second part and Appendix 1 are dedicated to more precise definitions and examples of complexity.

History and Overview,History and Philosophy of Physics
Ideas Behind Kolmogorov Complexity and Related Kolmogorov's Ideas about The Physical Impossibility of Large Integers: How Their Formalization Can Help in Foundations of Physics

Isaak A. Kunin,Vladik Kreinovich,Yuri A. Kuperin

DOI: https://doi.org/10.48550/arXiv.physics/0105038

2001-05-14

Abstract:In addition to the equations, physicists use the following additional difficult-to-formalize property: that the initial conditions and the value of the parameters must not be abnormal. We will describe a natural formalization of this property, and show that this formalization in good accordance with theoretical physics. At present, this formalization has been mainly applied to the foundations of physics. However, potentially, more practical applications are possible.

General Physics
Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Shengtian Yang,Thomas Honold,Zhaoyang Zhang

DOI: https://doi.org/10.1109/wcsp.2016.7752489

2016-01-01

Abstract:A general approach is established for deriving one-shot performance bounds for information-theoretic problems on general alphabets beyond countable alphabets. It is mainly based on the quantization idea and a novel form of "likelihood ratio". As an example, one-shot lower and upper bounds for random number generation from correlated sources on general alphabets are derived.
Theory and Applications of Probabilistic Kolmogorov Complexity

Zhenjian Lu,Igor C. Oliveira

DOI: https://doi.org/10.48550/arXiv.2205.14718

2022-05-29

Computational Complexity

Abstract:Diverse applications of Kolmogorov complexity to learning [CIKK16], circuit complexity [OPS19], cryptography [LP20], average-case complexity [Hir21], and proof search [Kra22] have been discovered in recent years. Since the running time of algorithms is a key resource in these fields, it is crucial in the corresponding arguments to consider time-bounded variants of Kolmogorov complexity. While fruitful interactions between time-bounded Kolmogorov complexity and different areas of theoretical computer science have been known for quite a while (e.g., [Sip83, Ko91, ABK+06, AF09], to name a few), the aforementioned results have led to a renewed interest in this topic. The theory of Kolmogorov complexity is well understood, but many useful results and properties of Kolmogorov complexity are not known to hold in time-bounded settings. This creates technical difficulties or leads to conditional results when applying methods from time-bounded Kolmogorov complexity to algorithms and complexity theory. Perhaps even more importantly, in many cases it is necessary to consider randomised algorithms. Since random strings have high complexity, the classical theory of time-bounded Kolmogorov complexity might be inappropriate in such contexts. To mitigate these issues and develop a theory of time-bounded Kolmogorov complexity that survives in the setting of randomised computations, some recent papers [Oli19, LO21, LOS21, GKLO22, LOZ22] have explored probabilistic notions of time-bounded Kolmogorov complexity, such as $\mathsf{rKt}$ complexity [Oli19], $\mathsf{rK}^t$ complexity [LOS21], and $\mathsf{pK}^t$ complexity [GKLO22]. These measures consider different ways of encoding an object via a probabilistic representation. In this survey, we provide an introduction to probabilistic time-bounded Kolmogorov complexity and its applications, highlighting many open problems and research directions.
An operational characterization of the notion of probability by algorithmic randomness II: Discrete probability spaces

Kohtaro Tadaki

DOI: https://doi.org/10.48550/arXiv.1909.02854

2019-08-30

Abstract:The notion of probability plays an important role in almost all areas of science and technology. In modern mathematics, however, probability theory means nothing other than measure theory, and the operational characterization of the notion of probability is not established yet. In this paper, based on the toolkit of algorithmic randomness we present an operational characterization of the notion of probability, called an ensemble, for general discrete probability spaces whose sample space is countably infinite. Algorithmic randomness, also known as algorithmic information theory, is a field of mathematics which enables us to consider the randomness of an individual infinite sequence. We use an extension of Martin-Loef randomness with respect to a generalized Bernoulli measure over the Baire space, in order to present the operational characterization. In our former work [K. Tadaki, <a class="link-https" data-arxiv-id="1611.06201" href="https://arxiv.org/abs/1611.06201">arXiv:1611.06201</a>], we developed an operational characterization of the notion of probability for an arbitrary finite probability space, i.e., a probability space whose sample space is a finite set. We then gave a natural operational characterization of the notion of conditional probability in terms of ensemble for a finite probability space, and gave equivalent characterizations of the notion of independence between two events based on it. Furthermore, we gave equivalent characterizations of the notion of independence of an arbitrary number of events/random variables in terms of ensembles for finite probability spaces. In particular, we showed that the independence between events/random variables is equivalent to the independence in the sense of van Lambalgen's Theorem, in the case where the underlying finite probability space is computable. In this paper, we show that we can certainly extend these results over general discrete probability spaces whose sample space is countably infinite.

Probability

Algorithmic Information Theory and Foundations of Probability

Foundations of Probability Theory for AI - The Application of Algorithmic Probability to Problems in Artificial Intelligence

Algorithmic statistics revisited

Foundations of abstract probability theory

Probabilistic foundations of quantum mechanics and quantum information

On the Algorithmic Information Between Probabilities

Diverse Consequences of Algorithmic Probability

Foundations of non-commutative probability theory (Extended abstract)

Introduction to foundations of probability and randomness (for students in physics), Lectures given at the Institute of Quantum Optics and Quantum Information, Austrian Academy of Science, Lecture-1: Kolmogorov and von Mises

Methods of Information Theory and Algorithmic Complexity for Network Biology

Information, complexity, brains and reality (Kolmogorov Manifesto)

Kolmogorov complexity in perspective

Epistemology as Information Theory: From Leibniz to Omega

Quantum Algorithmic Entropy

Is Randomness "Native" to Computer Science?

Information vs Dimension -- an Algorithmic Perspective

Kolmogorov complexity as a hidden factor of scientific discourse: from Newton's law to data mining

Ideas Behind Kolmogorov Complexity and Related Kolmogorov's Ideas about The Physical Impossibility of Large Integers: How Their Formalization Can Help in Foundations of Physics

Beyond Countable Alphabets: an Extension of the Information-Spectrum Approach.

Theory and Applications of Probabilistic Kolmogorov Complexity

An operational characterization of the notion of probability by algorithmic randomness II: Discrete probability spaces