Notes on a Path to AI Assistance in Mathematical Reasoning

Alex Kontorovich

2023-10-04

Abstract:These informal notes are based on the author's lecture at the National Academies of Science, Engineering, and Mathematics workshop on "AI to Assist Mathematical Reasoning" in June 2023. The goal is to think through a path by which we might arrive at AI that is useful for the research mathematician.

History and Overview,Artificial Intelligence

What problem does this paper attempt to address?

Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems

Keyur Faldu,Amit Sheth,Prashant Kikani,Manas Gaur,Aditi Avasthi

DOI: https://doi.org/10.48550/arXiv.2111.05364

2021-10-29

Abstract:Mathematical reasoning would be one of the next frontiers for artificial intelligence to make significant progress. The ongoing surge to solve math word problems (MWPs) and hence achieve better mathematical reasoning ability would continue to be a key line of research in the coming time. We inspect non-neural and neural methods to solve math word problems narrated in a natural language. We also highlight the ability of these methods to be generalizable, mathematically reasonable, interpretable, and explainable. Neural approaches dominate the current state of the art, and we survey them highlighting three strategies to MWP solving: (1) direct answer generation, (2) expression tree generation for inferring answers, and (3) template retrieval for answer computation. Moreover, we discuss technological approaches, review the evolution of intuitive design choices to solve MWPs, and examine them for mathematical reasoning ability. We finally identify several gaps that warrant the need for external knowledge and knowledge-infused learning, among several other opportunities in solving MWPs.

Computation and Language,Artificial Intelligence
AI for Mathematics: A Cognitive Science Perspective

Cedegao E. Zhang,Katherine M. Collins,Adrian Weller,Joshua B. Tenenbaum

2023-10-19

Abstract:Mathematics is one of the most powerful conceptual systems developed and used by the human species. Dreams of automated mathematicians have a storied history in artificial intelligence (AI). Rapid progress in AI, particularly propelled by advances in large language models (LLMs), has sparked renewed, widespread interest in building such systems. In this work, we reflect on these goals from a \textit{cognitive science} perspective. We call attention to several classical and ongoing research directions from cognitive science, which we believe are valuable for AI practitioners to consider when seeking to build truly human (or superhuman)-level mathematical systems. We close with open discussions and questions that we believe necessitate a multi-disciplinary perspective -- cognitive scientists working in tandem with AI researchers and mathematicians -- as we move toward better mathematical AI systems which not only help us push the frontier of the mathematics, but also offer glimpses into how we as humans are even capable of such great cognitive feats.

Neurons and Cognition,Artificial Intelligence
Computational principles of intelligence: learning and reasoning with neural networks

Abel Torres Montoya

DOI: https://doi.org/10.48550/arXiv.2012.09477

2020-12-17

Abstract:Despite significant achievements and current interest in machine learning and artificial intelligence, the quest for a theory of intelligence, allowing general and efficient problem solving, has done little progress. This work tries to contribute in this direction by proposing a novel framework of intelligence based on three principles. First, the generative and mirroring nature of learned representations of inputs. Second, a grounded, intrinsically motivated and iterative process for learning, problem solving and imagination. Third, an ad hoc tuning of the reasoning mechanism over causal compositional representations using inhibition rules. Together, those principles create a systems approach offering interpretability, continuous learning, common sense and more. This framework is being developed from the following perspectives: as a general problem solving method, as a human oriented tool and finally, as model of information processing in the brain.

Artificial Intelligence,Machine Learning
Analysing Mathematical Reasoning Abilities of Neural Models

David Saxton,Edward Grefenstette,Felix Hill,Pushmeet Kohli

DOI: https://doi.org/10.48550/arXiv.1904.01557

2019-04-03

Abstract:Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test splits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

Machine Learning
Mathematics, word problems, common sense, and artificial intelligence

Ernest Davis

DOI: https://doi.org/10.1090/bull/1828

2024-02-15

Bulletin of the American Mathematical Society

Abstract:The paper discusses the capacities and limitations of current artificial intelligence (AI) technology to solve word problems that combine elementary mathematics with commonsense reasoning. No existing AI systems can solve these reliably. We review three approaches that have been developed, using AI natural language technology: outputting the answer directly, outputting a computer program that solves the problem, and outputting a formalized representation that can be input to an automated theorem verifier. We review some benchmarks that have been developed to evaluate these systems and some experimental studies. We discuss the limitations of the existing technology at solving these kinds of problems. We argue that it is not clear whether these kinds of limitations will be important in developing AI technology for pure mathematical research, but that they will be important in applications of mathematics, and may well be important in developing programs capable of reading and understanding mathematical content written by humans.

mathematics
Advancing mathematics by guiding human intuition with AI

Alex Davies,Petar Veličković,Lars Buesing,Sam Blackwell,Daniel Zheng,Nenad Tomašev,Richard Tanburn,Peter Battaglia,Charles Blundell,András Juhász,Marc Lackenby,Geordie Williamson,Demis Hassabis,Pushmeet Kohli

DOI: https://doi.org/10.1038/s41586-021-04086-x

IF: 64.8

2021-12-01

Nature

Abstract:Abstract The practice of mathematics involves discovering patterns and using these to formulate and prove conjectures, resulting in theorems. Since the 1960s, mathematicians have used computers to assist in the discovery of patterns and formulation of conjectures 1 , most famously in the Birch and Swinnerton-Dyer conjecture 2 , a Millennium Prize Problem 3 . Here we provide examples of new fundamental results in pure mathematics that have been discovered with the assistance of machine learning—demonstrating a method by which machine learning can aid mathematicians in discovering new conjectures and theorems. We propose a process of using machine learning to discover potential patterns and relations between mathematical objects, understanding them with attribution techniques and using these observations to guide intuition and propose conjectures. We outline this machine-learning-guided framework and demonstrate its successful application to current research questions in distinct areas of pure mathematics, in each case showing how it led to meaningful mathematical contributions on important open problems: a new connection between the algebraic and geometric structure of knots, and a candidate algorithm predicted by the combinatorial invariance conjecture for symmetric groups 4 . Our work may serve as a model for collaboration between the fields of mathematics and artificial intelligence (AI) that can achieve surprising results by leveraging the respective strengths of mathematicians and machine learning.

multidisciplinary sciences
Mathematics Education in the Age of Artificial Intelligence

DOI: https://doi.org/10.1007/978-3-030-86909-0

2022-01-01

Abstract:This book highlights the contribution of artificial intelligence for mathematics education. It provides concrete ideas supported by mathematical work obtained through dynamic international collaboration, and discusses the flourishing of new mathematics in the contemporary world from a sustainable development perspective. Over the past thirty years, artificial intelligence has gradually infiltrated all facets of society. When it is deployed in interaction with the human designer or user, AI certainly raises new ethical questions. But as soon as it aims to augment intelligence in a kind of human-machine partnership, it goes to the heart of knowledge development and the very performance of work. The proposed themes and the sections of the book address original issues relating to the creation of AI milieus to work on mathematics, to the AI-supported learning of mathematics and to the coordination of « usual » paper/pencil techniques and « new » AI-aided educational working spaces. The authors of the book and the coordinators of each section are all established specialists in mathematics didactics, mathematics and computer science. In summary, this book is a must-read for everyone interested in the teaching and learning of mathematics, and it concerns the interaction between the human and the machine in both directions. It contains ideas, questions and inspiration that invite to take up the challenge of Artificial Intelligence contributing to Mathematical Human Learning.
Peano: Learning Formal Mathematical Reasoning

Gabriel Poesia,Noah D. Goodman

DOI: https://doi.org/10.1098/rsta.2022.0044

2022-11-29

Abstract:General mathematical reasoning is computationally undecidable, but humans routinely solve new problems. Moreover, discoveries developed over centuries are taught to subsequent generations quickly. What structure enables this, and how might that inform automated mathematical reasoning? We posit that central to both puzzles is the structure of procedural abstractions underlying mathematics. We explore this idea in a case study on 5 sections of beginning algebra on the Khan Academy platform. To define a computational foundation, we introduce Peano, a theorem-proving environment where the set of valid actions at any point is finite. We use Peano to formalize introductory algebra problems and axioms, obtaining well-defined search problems. We observe existing reinforcement learning methods for symbolic reasoning to be insufficient to solve harder problems. Adding the ability to induce reusable abstractions ("tactics") from its own solutions allows an agent to make steady progress, solving all problems. Furthermore, these abstractions induce an order to the problems, seen at random during training. The recovered order has significant agreement with the expert-designed Khan Academy curriculum, and second-generation agents trained on the recovered curriculum learn significantly faster. These results illustrate the synergistic role of abstractions and curricula in the cultural transmission of mathematics.

Artificial Intelligence
Mathematical reasoning and the computer

Kevin Buzzard

DOI: https://doi.org/10.1090/bull/1833

2024-02-15

Bulletin of the American Mathematical Society

Abstract:Computers have already changed the way that humans do mathematics: they enable us to compute efficiently. But will they soon be helping us to reason ? And will they one day start reasoning themselves? We give an overview of recent developments in neural networks, computer theorem provers, and large language models.

mathematics
AI Reasoning Systems: PAC and Applied Methods

Jeffrey Cheng

DOI: https://doi.org/10.48550/arXiv.1807.05054

IF: 14.4

2018-07-09

Artificial Intelligence

Abstract:Learning and logic are distinct and remarkable approaches to prediction. Machine learning has experienced a surge in popularity because it is robust to noise and achieves high performance; however, ML experiences many issues with knowledge transfer and extrapolation. In contrast, logic is easily intepreted, and logical rules are easy to chain and transfer between systems; however, inductive logic is brittle to noise. We then explore the premise of combining learning with inductive logic into AI Reasoning Systems. Specifically, we summarize findings from PAC learning (conceptual graphs, robust logics, knowledge infusion) and deep learning (DSRL, $\partial$ILP, DeepLogic) by reproducing proofs of tractability, presenting algorithms in pseudocode, highlighting results, and synthesizing between fields. We conclude with suggestions for integrated models by combining the modules listed above and with a list of unsolved (likely intractable) problems.
A Survey of Deep Learning for Mathematical Reasoning

Pan Lu,Liang Qiu,Wenhao Yu,Sean Welleck,Kai-Wei Chang

DOI: https://doi.org/10.48550/arXiv.2212.10535

IF: 14.4

2022-12-20

Artificial Intelligence

Abstract:Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain.
A Triumvirate of AI Driven Theoretical Discovery

Yang-Hui He

2024-05-30

Abstract:Recent years have seen the dramatic rise of the usage of AI algorithms in pure mathematics and fundamental sciences such as theoretical physics. This is perhaps counter-intuitive since mathematical sciences require the rigorous definitions, derivations, and proofs, in contrast to the experimental sciences which rely on the modelling of data with error-bars. In this Perspective, we categorize the approaches to mathematical discovery as "top-down", "bottom-up" and "meta-mathematics", as inspired by historical examples. We review some of the progress over the last few years, comparing and contrasting both the advances and the short-comings in each approach. We argue that while the theorist is in no way in danger of being replaced by AI in the near future, the hybrid of human expertise and AI algorithms will become an integral part of theoretical discovery.

History and Overview,High Energy Physics - Theory,History and Philosophy of Physics
A Moonshot for AI Oracles in the Sciences

Bryan Kaiser,Tailin Wu,Maike Sonnewald,Colin Thackray,Skylar Callis

2024-06-26

Abstract:Nobel laureate Philip Anderson and Elihu Abrahams once stated that, "even if machines did contribute to normal science, we see no mechanism by which they could create a Kuhnian revolution and thereby establish a new physical law." In this Perspective, we draw upon insights from the philosophies of science and artificial intelligence (AI) to propose necessary conditions of precisely such a mechanism for generating revolutionary mathematical theories. Recent advancements in AI suggest that satisfying the proposed necessary conditions by machines may be plausible; thus, our proposed necessary conditions also define a moonshot challenge. We also propose a heuristic definition of the intelligibility of mathematical theories to accelerate the development of machine theorists.

Artificial Intelligence,Computers and Society,History and Overview,Physics and Society
A Mathematical Approach to Constraining Neural Abstraction and the Mechanisms Needed to Scale to Higher-Order Cognition

Ananta Nair

DOI: https://doi.org/10.48550/arXiv.2108.05494

IF: 14.4

2021-08-12

Artificial Intelligence

Abstract:Artificial intelligence has made great strides in the last decade but still falls short of the human brain, the best-known example of intelligence. Not much is known of the neural processes that allow the brain to make the leap to achieve so much from so little beyond its ability to create knowledge structures that can be flexibly and dynamically combined, recombined, and applied in new and novel ways. This paper proposes a mathematical approach using graph theory and spectral graph theory, to hypothesize how to constrain these neural clusters of information based on eigen-relationships. This same hypothesis is hierarchically applied to scale up from the smallest to the largest clusters of knowledge that eventually lead to model building and reasoning.
Some thoughts on automation and mathematical research

Akshay Venkatesh

DOI: https://doi.org/10.1090/bull/1834

2024-02-16

Bulletin of the American Mathematical Society

Abstract:I discuss how mathematicians come to a shared notion of what is important, and how automated reasoning might affect that process.

mathematics
The Mathematics of Artificial Intelligence

Gitta Kutyniok

DOI: https://doi.org/10.48550/arXiv.2203.08890

2022-03-17

Abstract:We currently witness the spectacular success of artificial intelligence in both science and public life. However, the development of a rigorous mathematical foundation is still at an early stage. In this survey article, which is based on an invited lecture at the International Congress of Mathematicians 2022, we will in particular focus on the current "workhorse" of artificial intelligence, namely deep neural networks. We will present the main theoretical directions along with several exemplary results and discuss key open problems.

Machine Learning,History and Overview
Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes

Yezeng Chen,Zui Chen,Yi Zhou

2024-02-24

Abstract:Although large language models demonstrate emergent abilities in solving math word problems, there is a challenging task in complex multi-step mathematical reasoning tasks. To improve model performance on mathematical reasoning tasks, previous work has conducted supervised fine-tuning on open-source models by improving the quality and quantity of data. In this paper, we propose a novel approach, named Brain, to imitate human thought processes to enhance mathematical reasoning abilities, using the Frontal Lobe Model to generate plans, and then employing the Parietal Lobe Model to generate code and execute to obtain answers. First, we achieve SOTA performance in comparison with Code LLaMA 7B based models through this method. Secondly, we find that plans can be explicitly extracted from natural language, code, or formal language. Our code and data are publicly available at

Computation and Language,Artificial Intelligence,Machine Learning
A functional contextual, observer-centric, quantum mechanical, and neuro-symbolic approach to solving the alignment problem of artificial general intelligence: safe AI through intersecting computational psychological neuroscience and LLM architecture for emergent theory of mind

Darren J Edwards

DOI: https://doi.org/10.3389/fncom.2024.1395901

2024-08-08

Abstract:There have been impressive advancements in the field of natural language processing (NLP) in recent years, largely driven by innovations in the development of transformer-based large language models (LLM) that utilize "attention." This approach employs masked self-attention to establish (via similarly) different positions of tokens (words) within an inputted sequence of tokens to compute the most appropriate response based on its training corpus. However, there is speculation as to whether this approach alone can be scaled up to develop emergent artificial general intelligence (AGI), and whether it can address the alignment of AGI values with human values (called the alignment problem). Some researchers exploring the alignment problem highlight three aspects that AGI (or AI) requires to help resolve this problem: (1) an interpretable values specification; (2) a utility function; and (3) a dynamic contextual account of behavior. Here, a neurosymbolic model is proposed to help resolve these issues of human value alignment in AI, which expands on the transformer-based model for NLP to incorporate symbolic reasoning that may allow AGI to incorporate perspective-taking reasoning (i.e., resolving the need for a dynamic contextual account of behavior through deictics) as defined by a multilevel evolutionary and neurobiological framework into a functional contextual post-Skinnerian model of human language called "Neurobiological and Natural Selection Relational Frame Theory" (N-Frame). It is argued that this approach may also help establish a comprehensible value scheme, a utility function by expanding the expected utility equation of behavioral economics to consider functional contextualism, and even an observer (or witness) centric model for consciousness. Evolution theory, subjective quantum mechanics, and neuroscience are further aimed to help explain consciousness, and possible implementation within an LLM through correspondence to an interface as suggested by N-Frame. This argument is supported by the computational level of hypergraphs, relational density clusters, a conscious quantum level defined by QBism, and real-world applied level (human user feedback). It is argued that this approach could enable AI to achieve consciousness and develop deictic perspective-taking abilities, thereby attaining human-level self-awareness, empathy, and compassion toward others. Importantly, this consciousness hypothesis can be directly tested with a significance of approximately 5-sigma significance (with a 1 in 3.5 million probability that any identified AI-conscious observations in the form of a collapsed wave form are due to chance factors) through double-slit intent-type experimentation and visualization procedures for derived perspective-taking relational frames. Ultimately, this could provide a solution to the alignment problem and contribute to the emergence of a theory of mind (ToM) within AI.
Mathematical intuition, deep learning, and Robbins' problem

F. Thomas Bruss

2023-12-15

Abstract:{\bf Abstract.} The present article is an essay about mathematical intuition and Artificial intelligence (A.I.), followed by a guided excursion to a well-known open problem. It has two objectives. The first is to reconcile the way of thinking of a computer program as a sequence of mathematically defined instructions with what we face nowadays with newer developments. The second and major goal is to guide interested readers through the probabilistic intuition behind Robbins' problem and to show why A.I., and in particular Deep Learning, may contribute an essential part in its solution.

History and Overview,Probability

Notes on a Path to AI Assistance in Mathematical Reasoning

Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems

AI for Mathematics: A Cognitive Science Perspective

Computational principles of intelligence: learning and reasoning with neural networks

Analysing Mathematical Reasoning Abilities of Neural Models

Mathematics, word problems, common sense, and artificial intelligence

Advancing mathematics by guiding human intuition with AI

Mathematics Education in the Age of Artificial Intelligence

Peano: Learning Formal Mathematical Reasoning

Mathematical reasoning and the computer

AI Reasoning Systems: PAC and Applied Methods

A Survey of Deep Learning for Mathematical Reasoning

A Triumvirate of AI Driven Theoretical Discovery

A Moonshot for AI Oracles in the Sciences

A Mathematical Approach to Constraining Neural Abstraction and the Mechanisms Needed to Scale to Higher-Order Cognition

Some thoughts on automation and mathematical research

The Mathematics of Artificial Intelligence

Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes

A functional contextual, observer-centric, quantum mechanical, and neuro-symbolic approach to solving the alignment problem of artificial general intelligence: safe AI through intersecting computational psychological neuroscience and LLM architecture for emergent theory of mind

Mathematical intuition, deep learning, and Robbins' problem