Abstract:The mathematical formula is the human language to describe nature and is the essence of scientific research. Finding mathematical formulas from observational data is a major demand of scientific research and a major challenge of artificial intelligence. This area is called symbolic regression. Originally symbolic regression was often formulated as a combinatorial optimization problem and solved using GP or reinforcement learning algorithms. These two kinds of algorithms have strong noise robustness ability and good Versatility. However, inference time usually takes a long time, so the search efficiency is relatively low. Later, based on large-scale pre-training data proposed, such methods use a large number of synthetic data points and expression pairs to train a Generative Pre-Trained Transformer(GPT). Then this GPT can only need to perform one forward propagation to obtain the results, the advantage is that the inference speed is very fast. However, its performance is very dependent on the training data and performs poorly on data outside the training set, which leads to poor noise robustness and Versatility of such methods. So, can we combine the advantages of the above two categories of SR algorithms? In this paper, we propose \textbf{FormulaGPT}, which trains a GPT using massive sparse reward learning histories of reinforcement learning-based SR algorithms as training data. After training, the SR algorithm based on reinforcement learning is distilled into a Transformer. When new test data comes, FormulaGPT can directly generate a "reinforcement learning process" and automatically update the learning policy in context. Tested on more than ten datasets including SRBench, formulaGPT achieves the state-of-the-art performance in fitting ability compared with four baselines. In addition, it achieves satisfactory results in noise robustness, versatility, and inference efficiency.

Guiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities

Leveraging Grammar Induction for Language Understanding and Generation

Do Transformers know symbolic rules, and would we know if they did?

Automatic Rule Induction for Interpretable Semi-Supervised Learning

The Go Transformer: Natural Language Modeling for Game Play

Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically

Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

Syntax-informed Question Answering with Heterogeneous Graph Transformer

Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers

Transduce: learning transduction grammars for string transformation

Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

A Symbolic Rule Integration Framework with Logic Transformer for Inductive Relation Prediction

The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

Sneaking Syntax into Transformer Language Models with Tree Regularization

Natural Language Inference with Transformer Ensembles and Explainability Techniques

Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

Generative AI in the Era of Transformers: Revolutionizing Natural Language Processing with LLMs

SymbolicGPT: A Generative Transformer Model for Symbolic Regression

Injecting structural hints: Using language models to study inductive biases in language learning

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale