Towards Efficient Neurally-Guided Program Induction for ARC-AGI

Simon Ouellette

2024-11-13

Abstract:ARC-AGI is an open-world problem domain in which the ability to generalize out-of-distribution is a crucial quality. Under the program induction paradigm, we present a series of experiments that reveal the efficiency and generalization characteristics of various neurally-guided program induction approaches. The three paradigms we consider are Learning the grid space, Learning the program space, and Learning the transform space. We implement and experiment thoroughly on the first two, and retain the second one for ARC-AGI submission. After identifying the strengths and weaknesses of both of these approaches, we suggest the third as a potential solution, and run preliminary experiments.

Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper mainly explores how to improve the generalization ability of the program induction method in the open - world problem domain, especially in the ARC - AGI (Abstraction & Reasoning Corpus for Artificial General Intelligence) task. Specifically, the paper focuses on how to improve the generalization performance of the model on out - of - distribution (OOD) data.

#### Main problem background

1. **Differences between the open world and the closed world**:
- In the closed - world, the model can cover all possibilities through dense sampling and interpolation, and deep learning can usually achieve performance close to or surpassing that of humans.
- However, in the open world, because the range of possibilities is too large to be completely covered by dense sampling, interpolation - based techniques such as deep learning perform poorly in out - of - distribution generalization.

2. **Characteristics of ARC - AGI**:
- ARC - AGI is a problem domain specifically designed to challenge the model's out - of - distribution generalization ability. It uses a hidden test set that contains tasks that are qualitatively different from publicly available tasks, encouraging researchers to develop learning systems that can perform well on out - of - distribution data.

#### Main objectives of the paper

Through a series of experiments, the paper evaluates the efficiency and generalization characteristics of three different paradigms of neural - guided program induction methods, aiming to find a solution that can effectively generalize in the ARC - AGI task. These three paradigms are:

1. **Learning the Grid Space (LGS)**:
- The goal is to learn a possible ARC - AGI grid - space model under a specific DSL, and guide program construction by estimating the similarity between two different grids.

2. **Learning the Program Space (LPS)**:
- Train a Transformer model to output programs that conform to a predefined grammar (DSL), and use a search algorithm to enumerate and validate valid programs in this space.

3. **Learning the Transformation Space (LTS)**:
- Combine the advantages of LGS and LPS, and try to more effectively guide program construction by predicting the probability distribution of the next DSL token given the intermediate program state and the target grid.

#### Main contributions

1. **For the first time, analyze the characteristics of the program induction method based on enumerating the program probability space in terms of out - of - distribution generalization**.
2. **Provide the preliminary results of the LGS method on the ARC - AGI task and an analysis of its advantages and disadvantages**.
3. **Propose a probabilistic program enumeration search algorithm based on the Transformer autoregressive token sequence and conduct a detailed analysis**.
4. **Outline a new LTS method, aiming to combine the advantages of the previous two methods, and support this hypothesis through preliminary experiments**.

Through these studies, the paper hopes to provide valuable insights for future research to improve the out - of - distribution generalization ability of deep - learning models in open - world problems.