Abstract:Inductive reasoning is a core problem-solving capacity: humans can identify underlying principles from a few examples, which robustly generalize to novel scenarios. Recent work evaluates large language models (LLMs) on inductive reasoning tasks by directly prompting them yielding "in context learning." This works well for straightforward inductive tasks but performs poorly on complex tasks such as the Abstraction and Reasoning Corpus (ARC). In this work, we propose to improve the inductive reasoning ability of LLMs by generating explicit hypotheses at multiple levels of abstraction: we prompt the LLM to propose multiple abstract hypotheses about the problem, in natural language, then implement the natural language hypotheses as concrete Python programs. These programs can be verified by running on observed examples and generalized to novel inputs. To reduce the hypothesis search space, we explore steps to filter the set of hypotheses to implement: we either ask the LLM to summarize them into a smaller set of hypotheses or ask human annotators to select a subset. We verify our pipeline's effectiveness on the ARC visual inductive reasoning benchmark, its variant 1D-ARC, string transformation dataset SyGuS, and list transformation dataset List Functions. On a random 100-problem subset of ARC, our automated pipeline using LLM summaries achieves 30% accuracy, outperforming the direct prompting baseline (accuracy of 17%). With the minimal human input of selecting from LLM-generated candidates, performance is boosted to 33%. Our ablations show that both abstract hypothesis generation and concrete program representations benefit LLMs on inductive reasoning tasks.

Effectiveness of Positive Hypothesis Testing in Inductive and Deductive Rule Learning

Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models

Observable Changes of Hypotheses under Positive Reinforcement

Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement

Syllogistic reasoning and belief-bias inhibition in school children: evidence from a negative priming paradigm.

Intelligent Feedback on Hypothesis Testing

Do smart people have better intuitions?

The Impact of Looking Further Ahead: A Comparison of Two Data-driven Unsolicited Hint Types on Performance in an Intelligent Data-driven Logic Tutor

When Problem Solving Followed by Instruction Works: Evidence for Productive Failure

Exemplar learners and rule learners: Stable tendencies or malleable preferences?

Efficiency of Learning from Proof Blocks Versus Writing Proofs

Influence of Positive Affect upon Creative Thinking and Problem Solving in Children

In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings

Étude des stratégies de raisonnement causal dans l'estimation de la probabilité diagnostique à travers un paradigme expérimental de production de règle

The Learning Power of Testing

HGR 2.0: A New Rule Induction Algorithm

Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning

Opposites in Reasoning Processes: Do We Use Them More Than We Think, but Less Than We Could?

Hypothesis Search: Inductive Reasoning with Language Models

Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs

Selective attention in hypothesis-driven data analysis