Exploration with Principles for Diverse AI Supervision

Hao Liu,Matei Zaharia,Pieter Abbeel

2023-11-23

Abstract:Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. Even state-of-the-art AI models like ChatGPT depend on fine-tuning through human demonstrations, demanding extensive human input and domain expertise. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. To address this limitation, we propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data. Drawing inspiration from unsupervised reinforcement learning (RL) pretraining, EAI achieves exploration within the natural language space. We accomplish this by harnessing large language models to assess the novelty of generated content. Our approach employs two key components: an actor that generates novel content following exploration principles and a critic that evaluates the generated content, offering critiques to guide the actor. Empirical evaluations demonstrate that EAI significantly boosts model performance on complex reasoning tasks, addressing the limitations of human-intensive supervision.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the issue of current AI systems (especially large Transformer models) being overly dependent on human supervision during training. Specifically, although large Transformer models based on next-word prediction (such as ChatGPT) have made groundbreaking progress in the field of AI, these models still require fine-tuning through human demonstrations, which demands a significant amount of expert human input. This strong reliance on human supervision limits AI innovation and development.

To solve this problem, the paper proposes a new paradigm called "Exploratory AI" (EAI), which aims to utilize AI models to autonomously generate high-quality training data, thereby reducing the need for human supervision. EAI is inspired by unsupervised reinforcement learning pretraining methods and achieves exploration within the natural language space. Its core mechanism includes a generator (referred to as the "actor," responsible for generating novel content) and an evaluator (referred to as the "critic," responsible for assessing the novelty and correctness of the generated content and providing feedback to guide the generator). In this way, EAI can autonomously generate diverse training data, thereby improving the model's performance on complex reasoning tasks.

The main contributions of the paper can be summarized as follows:
1. Proposing EAI as a new method that leverages the capabilities of large language models to autonomously generate diverse, high-quality training data, eliminating the dependence on human-intensive supervision.
2. Validating the effectiveness of EAI through a series of experiments, demonstrating that it significantly outperforms previous methods, including human supervision and AI supervision baselines, in mathematical reasoning benchmarks.

In short, the goal of the paper is to develop a new training method that allows AI models to largely eliminate the dependence on human supervision, autonomously generate diverse high-quality training data, and thereby improve their performance on complex tasks.