Falsifiable implies Learnable

David Balduzzi

DOI: https://doi.org/10.48550/arXiv.1408.6618

IF: 5.414

2014-08-28

Machine Learning

Abstract:The paper demonstrates that falsifiability is fundamental to learning. We prove the following theorem for statistical learning and sequential prediction: If a theory is falsifiable then it is learnable -- i.e. admits a strategy that predicts optimally. An analogous result is shown for universal induction.

What problem does this paper attempt to address?

The Model Theory of Falsification

Reid Dale

DOI: https://doi.org/10.48550/arXiv.2209.12067

2022-09-25

Abstract:This paper is concerned with the question of when a theory is refutable with certainty on the basis of sequence of primitive observations. Beginning with the simple definition of falsifiability as the ability to be refuted by some finite collection of observations, I assess the literature on falsification and its descendants within the context of the dividing lines of contemporary model theory. The static case is broadly concerned with the question of how much of a theory can be subjected to falsifying experiments. In much of the literature, this question is tied up with whether the theory in question is axiomatizable by a collection of universal first-order sentences. I argue that this is too narrow a conception of falsification by demonstrating that a natural class of theories of distinct model-theoretic interest -- so-called NIP theories -- are themselves highly falsifiable.

Logic
The Combinatorics of Falsification and Hypothesis Testing

Reid Dale

DOI: https://doi.org/10.48550/arXiv.2209.12066

2022-09-25

Abstract:The present paper is concerned with the question of how falsifiable a single proposition is in the short and long run. Formal Learning theorists such as Schulte and Juhl have argued that long-run falsifiability is characterized by the topological notion of nowhere density in a suitable topological space. I argue that the short-run falsifiability of a hypothesis is in turn characterized by the VC finiteness of the hypothesis. Crucially, VC finite hypotheses correspond precisely to definable sets in NIP structures. I end the chapter by giving rigorous foundations for Mayo's conception of severe testing by way of a combinatorial, non-probabilistic notion of surprise. VC finite hypotheses again appear as the hypotheses with guaranteed short-run surprise bounds. Therefore, NIP theories and VC finite hypotheses capture the notion of short-run falsifiability.

Logic
From Undecidability of Non-Triviality and Finiteness to Undecidability of Learnability

Matthias C. Caro

DOI: https://doi.org/10.1016/j.ijar.2023.109057

2023-10-17

Abstract:Machine learning researchers and practitioners steadily enlarge the multitude of successful learning models. They achieve this through in-depth theoretical analyses and experiential heuristics. However, there is no known general-purpose procedure for rigorously evaluating whether newly proposed models indeed successfully learn from data. We show that such a procedure cannot exist. For PAC binary classification, uniform and universal online learning, and exact learning through teacher-learner interactions, learnability is in general undecidable, both in the sense of independence of the axioms in a formal system and in the sense of uncomputability. Our proofs proceed via computable constructions that encode the consistency problem for formal systems and the halting problem for Turing machines into whether certain function classes are trivial/finite or highly complex, which we then relate to whether these classes are learnable via established characterizations of learnability through complexity measures. Our work shows that undecidability appears in the theoretical foundations of artificial intelligence: There is no one-size-fits-all algorithm for deciding whether a machine learning model can be successful. We cannot in general automatize the process of assessing new learning models.

Computational Complexity,Machine Learning,Logic
Redeeming Falsifiability?

Mark Whitmeyer,Kun Zhang

DOI: https://doi.org/10.48550/arXiv.2303.15723

2023-03-28

Abstract:We revisit Popper's falsifiability criterion. A tester hires a potential expert to produce a theory, offering payments contingent on the observed performance of the theory. We argue that if the informed expert can acquire additional information, falsifiability does have the power to identify worthless theories.

Theoretical Economics
Falsification and consciousness

Johannes Kleiner,Erik Hoel

DOI: https://doi.org/10.1093/nc/niab001

2021-04-28

Abstract:The search for a scientific theory of consciousness should result in theories that are falsifiable. However, here we show that falsification is especially problematic for theories of consciousness. We formally describe the standard experimental setup for testing these theories. Based on a theory's application to some physical system, such as the brain, testing requires comparing a theory's predicted experience (given some internal observables of the system like brain imaging data) with an inferred experience (using report or behavior). If there is a mismatch between inference and prediction, a theory is falsified. We show that if inference and prediction are independent, it follows that any minimally informative theory of consciousness is automatically falsified. This is deeply problematic since the field's reliance on report or behavior to infer conscious experiences implies such independence, so this fragility affects many contemporary theories of consciousness. Furthermore, we show that if inference and prediction are strictly dependent, it follows that a theory is unfalsifiable. This affects theories which claim consciousness to be determined by report or behavior. Finally, we explore possible ways out of this dilemma.

Neurons and Cognition
Realizable Learning is All You Need

Max Hopkins,Daniel M. Kane,Shachar Lovett,Gaurav Mahajan

DOI: https://doi.org/10.46298/theoretics.24.2

2024-02-03

Abstract:The equivalence of realizable and agnostic learnability is a fundamental phenomenon in learning theory. With variants ranging from classical settings like PAC learning and regression to recent trends such as adversarially robust learning, it's surprising that we still lack a unified theory; traditional proofs of the equivalence tend to be disparate, and rely on strong model-specific assumptions like uniform convergence and sample compression. In this work, we give the first model-independent framework explaining the equivalence of realizable and agnostic learnability: a three-line blackbox reduction that simplifies, unifies, and extends our understanding across a wide variety of settings. This includes models with no known characterization of learnability such as learning with arbitrary distributional assumptions and more general loss functions, as well as a host of other popular settings such as robust learning, partial learning, fair learning, and the statistical query model. More generally, we argue that the equivalence of realizable and agnostic learning is actually a special case of a broader phenomenon we call property generalization: any desirable property of a learning algorithm (e.g. noise tolerance, privacy, stability) that can be satisfied over finite hypothesis classes extends (possibly in some variation) to any learnable hypothesis class.

Machine Learning
Comparing Differentiable Logics for Learning Systems: A Research Preview

Thomas Flinkow,Barak A. Pearlmutter,Rosemary Monahan

DOI: https://doi.org/10.4204/EPTCS.395.3

2023-11-16

Abstract:Extensive research on formal verification of machine learning (ML) systems indicates that learning from data alone often fails to capture underlying background knowledge. A variety of verifiers have been developed to ensure that a machine-learnt model satisfies correctness and safety properties, however, these verifiers typically assume a trained network with fixed weights. ML-enabled autonomous systems are required to not only detect incorrect predictions, but should also possess the ability to self-correct, continuously improving and adapting. A promising approach for creating ML models that inherently satisfy constraints is to encode background knowledge as logical constraints that guide the learning process via so-called differentiable logics. In this research preview, we compare and evaluate various logics from the literature in weakly-supervised contexts, presenting our findings and highlighting open problems for future work. Our experimental results are broadly consistent with results reported previously in literature; however, learning with differentiable logics introduces a new hyperparameter that is difficult to tune and has significant influence on the effectiveness of the logics.

Logic in Computer Science,Artificial Intelligence,Machine Learning
Learning to falsify automated driving vehicles with prior knowledge

Andrea Favrin,Vladislav Nenchev,Angelo Cenedese

DOI: https://doi.org/10.48550/arXiv.2101.10377

2021-01-26

Abstract:While automated driving technology has achieved a tremendous progress, the scalable and rigorous testing and verification of safe automated and autonomous driving vehicles remain challenging. This paper proposes a learning-based falsification framework for testing the implementation of an automated or self-driving function in simulation. We assume that the function specification is associated with a violation metric on possible scenarios. Prior knowledge is incorporated to limit the scenario parameter variance and in a model-based falsifier to guide and improve the learning process. For an exemplary adaptive cruise controller, the presented framework yields non-trivial falsifying scenarios with higher reward, compared to scenarios obtained by purely learning-based or purely model-based falsification approaches.

Robotics,Artificial Intelligence,Systems and Control
A Farewell to Falsifiability

Douglas Scott,Ali Frolop,Ali Narimani,Andrei Frolov

DOI: https://doi.org/10.48550/arXiv.1504.00108

2015-04-01

Abstract:Some of the most obviously correct physical theories - namely string theory and the multiverse - make no testable predictions, leading many to question whether we should accept something as scientific even if it makes no testable predictions and hence is not refutable. However, some far-thinking physicists have proposed instead that we should give up on the notion of Falsifiability itself. We endorse this suggestion but think it does not go nearly far enough. We believe that we should also dispense with other outdated ideas, such as Fidelity, Frugality, Factuality and other "F" words. And we quote a lot of famous people to support this view.

Cosmology and Nongalactic Astrophysics,High Energy Physics - Theory
When Is Inductive Inference Possible?

Zhou Lu

2024-09-26

Abstract:Can a physicist make only a finite number of errors in the eternal quest to uncover the law of nature? This millennium-old philosophical problem, known as inductive inference, lies at the heart of epistemology. Despite its significance to understanding human reasoning, a rigorous justification of inductive inference has remained elusive. At a high level, inductive inference asks whether one can make at most finite errors amidst an infinite sequence of observations, when deducing the correct hypothesis from a given hypothesis class. Historically, the only theoretical guarantee has been that if the hypothesis class is countable, inductive inference is possible, as exemplified by Solomonoff induction for learning Turing machines. In this paper, we provide a tight characterization of inductive inference by establishing a novel link to online learning theory. As our main result, we prove that inductive inference is possible if and only if the hypothesis class is a countable union of online learnable classes, potentially with an uncountable size, no matter the observations are adaptively chosen or iid sampled. Moreover, the same condition is also sufficient and necessary in the agnostic setting, where any hypothesis class meeting this criterion enjoys an $\tilde{O}(\sqrt{T})$ regret bound for any time step $T$, while others require an arbitrarily slow rate of regret. Our main technical tool is a novel non-uniform online learning framework, which may be of independent interest.

Machine Learning
Comparing Differentiable Logics for Learning with Logical Constraints

Thomas Flinkow,Barak A. Pearlmutter,Rosemary Monahan

2024-07-04

Abstract:Extensive research on formal verification of machine learning systems indicates that learning from data alone often fails to capture underlying background knowledge such as specifications implicitly available in the data. Various neural network verifiers have been developed to ensure that a machine-learnt model satisfies correctness and safety properties, however, they typically assume a trained network with fixed weights. A promising approach for creating machine learning models that inherently satisfy constraints after training is to encode background knowledge as explicit logical constraints that guide the learning process via so-called differentiable logics. In this paper, we experimentally compare and evaluate various logics from the literature, presenting our findings and highlighting open problems for future work.

Logic in Computer Science
Falsifiability of Isolated Spacetime Regions

Roberto C. Alamino

DOI: https://doi.org/10.48550/arXiv.1907.02855

2019-07-05

Abstract:In this work it is pointed out that some physical theories, even being themselves falsifiable, predict the existence of regions of spacetime which are not falsifiable with relation to each other due to their impossibility of mutually exchanging information as, for instance, before and after the event horizon of black holes. If we require scientific theories to be falsifiable, an isolated region should be discarded from scientific models developed by observers in other regions. Here it is proposed that their existence can satisfy a weaker falsifiability condition, here called conditional asymptotic provability, which extend scientific reasoning through Bayesian inference. Limitations and some epistemic consequences of this proposal are discussed.

History and Philosophy of Physics,General Relativity and Quantum Cosmology
Measurability in the Fundamental Theorem of Statistical Learning

Lothar Sebastian Krapp,Laura Wirth

2024-10-14

Abstract:The Fundamental Theorem of Statistical Learning states that a hypothesis space is PAC learnable if and only if its VC dimension is finite. For the agnostic model of PAC learning, the literature so far presents proofs of this theorem that often tacitly impose several measurability assumptions on the involved sets and functions. We scrutinize these proofs from a measure-theoretic perspective in order to extract the assumptions needed for a rigorous argument. This leads to a sound statement as well as a detailed and self-contained proof of the Fundamental Theorem of Statistical Learning in the agnostic setting, showcasing the minimal measurability requirements needed. We then discuss applications in Model Theory, considering NIP and o-minimal structures. Our main theorem presents sufficient conditions for the PAC learnability of hypothesis spaces defined over o-minimal expansions of the reals.

Machine Learning,Logic in Computer Science,Logic,Probability
Computable learning of natural hypothesis classes

Matthew Harrison-Trainor,Syed Akbari

2024-07-30

Abstract:This paper is about the recent notion of computably probably approximately correct learning, which lies between the statistical learning theory where there is no computational requirement on the learner and efficient PAC where the learner must be polynomially bounded. Examples have recently been given of hypothesis classes which are PAC learnable but not computably PAC learnable, but these hypothesis classes are unnatural or non-canonical in the sense that they depend on a numbering of proofs, formulas, or programs. We use the on-a-cone machinery from computability theory to prove that, under mild assumptions such as that the hypothesis class can be computably listable, any natural hypothesis class which is learnable must be computably learnable. Thus the counterexamples given previously are necessarily unnatural.

Machine Learning,Logic
Testably Learning Polynomial Threshold Functions

Lucas Slot,Stefan Tiegel,Manuel Wiedmer

2024-06-10

Abstract:Rubinfeld & Vasilyan recently introduced the framework of testable learning as an extension of the classical agnostic model. It relaxes distributional assumptions which are difficult to verify by conditions that can be checked efficiently by a tester. The tester has to accept whenever the data truly satisfies the original assumptions, and the learner has to succeed whenever the tester accepts. We focus on the setting where the tester has to accept standard Gaussian data. There, it is known that basic concept classes such as halfspaces can be learned testably with the same time complexity as in the (distribution-specific) agnostic model. In this work, we ask whether there is a price to pay for testably learning more complex concept classes. In particular, we consider polynomial threshold functions (PTFs), which naturally generalize halfspaces. We show that PTFs of arbitrary constant degree can be testably learned up to excess error $\varepsilon > 0$ in time $n^{\mathrm{poly}(1/\varepsilon)}$. This qualitatively matches the best known guarantees in the agnostic model. Our results build on a connection between testable learning and fooling. In particular, we show that distributions that approximately match at least $\mathrm{poly}(1/\varepsilon)$ moments of the standard Gaussian fool constant-degree PTFs (up to error $\varepsilon$). As a secondary result, we prove that a direct approach to show testable learning (without fooling), which was successfully used for halfspaces, cannot work for PTFs.

Machine Learning,Data Structures and Algorithms
Optimizing Falsification for Learning-Based Control Systems: A Multi-Fidelity Bayesian Approach

Zahra Shahrooei,Mykel J. Kochenderfer,Ali Baheri

2024-09-12

Abstract:Testing controllers in safety-critical systems is vital for ensuring their safety and preventing failures. In this paper, we address the falsification problem within learning-based closed-loop control systems through simulation. This problem involves the identification of counterexamples that violate system safety requirements and can be formulated as an optimization task based on these requirements. Using full-fidelity simulator data in this optimization problem can be computationally expensive. To improve efficiency, we propose a multi-fidelity Bayesian optimization falsification framework that harnesses simulators with varying levels of accuracy. Our proposed framework can transition between different simulators and establish meaningful relationships between them. Through multi-fidelity Bayesian optimization, we determine both the optimal system input likely to be a counterexample and the appropriate fidelity level for assessment. We evaluated our approach across various Gym environments, each featuring different levels of fidelity. Our experiments demonstrate that multi-fidelity Bayesian optimization is more computationally efficient than full-fidelity Bayesian optimization and other baseline methods in detecting counterexamples. A Python implementation of the algorithm is available at <a class="link-external link-https" href="https://github.com/SAILRIT/MFBO_Falsification" rel="external noopener nofollow">this https URL</a>.

Systems and Control,Machine Learning
Is there an Elegant Universal Theory of Prediction?

Shane Legg

DOI: https://doi.org/10.48550/arXiv.cs/0606070

IF: 14.4

2006-06-14

Artificial Intelligence

Abstract:Solomonoff's inductive learning model is a powerful, universal and highly elegant theory of sequence prediction. Its critical flaw is that it is incomputable and thus cannot be used in practice. It is sometimes suggested that it may still be useful to help guide the development of very general and powerful theories of prediction which are computable. In this paper it is shown that although powerful algorithms exist, they are necessarily highly complex. This alone makes their theoretical analysis problematic, however it is further shown that beyond a moderate level of complexity the analysis runs into the deeper problem of Goedel incompleteness. This limits the power of mathematics to analyse and study prediction algorithms, and indeed intelligent systems in general.
Replicability and stability in learning

Zachary Chase,Shay Moran,Amir Yehudayoff

2023-04-13

Abstract:Replicability is essential in science as it allows us to validate and verify research findings. Impagliazzo, Lei, Pitassi and Sorrell (`22) recently initiated the study of replicability in machine learning. A learning algorithm is replicable if it typically produces the same output when applied on two i.i.d. inputs using the same internal randomness. We study a variant of replicability that does not involve fixing the randomness. An algorithm satisfies this form of replicability if it typically produces the same output when applied on two i.i.d. inputs (without fixing the internal randomness). This variant is called global stability and was introduced by Bun, Livni and Moran ('20) in the context of differential privacy. Impagliazzo et al. showed how to boost any replicable algorithm so that it produces the same output with probability arbitrarily close to 1. In contrast, we demonstrate that for numerous learning tasks, global stability can only be accomplished weakly, where the same output is produced only with probability bounded away from 1. To overcome this limitation, we introduce the concept of list replicability, which is equivalent to global stability. Moreover, we prove that list replicability can be boosted so that it is achieved with probability arbitrarily close to 1. We also describe basic relations between standard learning-theoretic complexity measures and list replicable numbers. Our results, in addition, imply that besides trivial cases, replicable algorithms (in the sense of Impagliazzo et al.) must be randomized. The proof of the impossibility result is based on a topological fixed-point theorem. For every algorithm, we are able to locate a "hard input distribution" by applying the Poincaré-Miranda theorem in a related topological setting. The equivalence between global stability and list replicability is algorithmic.

Machine Learning,Combinatorics
Credal Learning Theory

Michele Caprio,Maryam Sultana,Eleni Elia,Fabio Cuzzolin

2024-05-03

Abstract:Statistical learning theory is the foundation of machine learning, providing theoretical bounds for the risk of models learnt from a (single) training set, assumed to issue from an unknown probability distribution. In actual deployment, however, the data distribution may (and often does) vary, causing domain adaptation/generalization issues. In this paper we lay the foundations for a `credal' theory of learning, using convex sets of probabilities (credal sets) to model the variability in the data-generating distribution. Such credal sets, we argue, may be inferred from a finite sample of training sets. Bounds are derived for the case of finite hypotheses spaces (both assuming realizability or not) as well as infinite model spaces, which directly generalize classical results.

Machine Learning,Artificial Intelligence

Falsifiable implies Learnable

The Model Theory of Falsification

The Combinatorics of Falsification and Hypothesis Testing

From Undecidability of Non-Triviality and Finiteness to Undecidability of Learnability

Redeeming Falsifiability?

Falsification and consciousness

Realizable Learning is All You Need

Comparing Differentiable Logics for Learning Systems: A Research Preview

Learning to falsify automated driving vehicles with prior knowledge

A Farewell to Falsifiability

When Is Inductive Inference Possible?

Comparing Differentiable Logics for Learning with Logical Constraints

Falsifiability of Isolated Spacetime Regions

Measurability in the Fundamental Theorem of Statistical Learning

Computable learning of natural hypothesis classes

Testably Learning Polynomial Threshold Functions

Optimizing Falsification for Learning-Based Control Systems: A Multi-Fidelity Bayesian Approach

Is there an Elegant Universal Theory of Prediction?

Replicability and stability in learning

Credal Learning Theory