Abstract:Can a physicist make only a finite number of errors in the eternal quest to uncover the law of nature? This millennium-old philosophical problem, known as inductive inference, lies at the heart of epistemology. Despite its significance to understanding human reasoning, a rigorous justification of inductive inference has remained elusive. At a high level, inductive inference asks whether one can make at most finite errors amidst an infinite sequence of observations, when deducing the correct hypothesis from a given hypothesis class. Historically, the only theoretical guarantee has been that if the hypothesis class is countable, inductive inference is possible, as exemplified by Solomonoff induction for learning Turing machines. In this paper, we provide a tight characterization of inductive inference by establishing a novel link to online learning theory. As our main result, we prove that inductive inference is possible if and only if the hypothesis class is a countable union of online learnable classes, potentially with an uncountable size, no matter the observations are adaptively chosen or iid sampled. Moreover, the same condition is also sufficient and necessary in the agnostic setting, where any hypothesis class meeting this criterion enjoys an $\tilde{O}(\sqrt{T})$ regret bound for any time step $T$, while others require an arbitrarily slow rate of regret. Our main technical tool is a novel non-uniform online learning framework, which may be of independent interest.

A General Inductive Principle for an Ordered Set and the Continual Induction

New Proof for Continual Induction and Its Application

Proving Inductive Theorems Using Construction Bases

The Instructor's Guide to Real Induction

First-Order Continuous Induction, and a Logical Study of Real Closed Fields

First-Order Continuous Induction and a Logical Study of Real Closed Fields

Independence and Induction in Reverse Mathematics

The Principle of Open Induction on Cantor space and the Approximate-Fan Theorem

An induction principle and pigeonhole principles for K-finite sets

Infrafiltration Theorem and Some Inductive Sequence of Models of Generalized Second-Order Dedekind Theory of Real Numbers With Exponentially Increasing Powers

An Axiomatic Foundation for the Logic of Inductive Generalization

Countability of Inductive Types Formalized in the Object-Logic Level

A Logic for Non-Monotone Inductive Definitions

Categorifying induction formulae via divergent series

Foundations of regular coinduction

A Logical Study of Some Common Principles of Inductive Definition and its Implications for Knowledge Representation

On Tao's "finitary" infinite pigeonhole principle

Automated Induction for Complex Data Structures

When Is Inductive Inference Possible?

Toward a Mathematical Theory of Inductive Inference

Analyzing the Collatz Conjecture Using the Mathematical Complete Induction Method