A Comparative Study on Reasoning Patterns of OpenAI's o1 Model

Siwei Wu,Zhongyuan Peng,Xinrun Du,Tuney Zheng,Minghao Liu,Jialong Wu,Jiachen Ma,Yizhi Li,Jian Yang,Wangchunshu Zhou,Qunshu Lin,Junbo Zhao,Zhaoxiang Zhang,Wenhao Huang,Ge Zhang,Chenghua Lin,J.H. Liu

2024-10-23

Abstract:Enabling Large Language Models (LLMs) to handle a wider range of complex tasks (e.g., coding, math) has drawn great attention from many researchers. As LLMs continue to evolve, merely increasing the number of model parameters yields diminishing performance improvements and heavy computational costs. Recently, OpenAI's o1 model has shown that inference strategies (i.e., Test-time Compute methods) can also significantly enhance the reasoning capabilities of LLMs. However, the mechanisms behind these methods are still unexplored. In our work, to investigate the reasoning patterns of o1, we compare o1 with existing Test-time Compute methods (BoN, Step-wise BoN, Agent Workflow, and Self-Refine) by using OpenAI's GPT-4o as a backbone on general reasoning benchmarks in three domains (i.e., math, coding, commonsense reasoning). Specifically, first, our experiments show that the o1 model has achieved the best performance on most datasets. Second, as for the methods of searching diverse responses (e.g., BoN), we find the reward models' capability and the search space both limit the upper boundary of these methods. Third, as for the methods that break the problem into many sub-problems, the Agent Workflow has achieved better performance than Step-wise BoN due to the domain-specific system prompt for planning better reasoning processes. Fourth, it is worth mentioning that we have summarized six reasoning patterns of o1, and provided a detailed analysis on several reasoning benchmarks.

Computation and Language

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to evaluate and compare the performance of large - language models (LLMs) on different reasoning tasks, especially the performance differences between OpenAI's o1 model and the existing Test - time Compute methods. Specifically, the paper focuses on the following points:

1. **Limitations of model parameter expansion**: As the number of model parameters increases, the performance improvement gradually decreases, and the computational cost becomes very expensive. Therefore, researchers have begun to explore other methods to improve the performance of LLMs.

2. **Effectiveness of test - time compute methods**: The paper explores the effects of different test - time compute methods (such as Best - of - N, Step - wise BoN, Agent Workflow, and Self - Refine) on improving the reasoning ability of LLMs. These methods enhance the model's performance by using additional computational resources during the reasoning stage.

3. **Reasoning mechanism of the o1 model**: The paper analyzes in detail the performance of OpenAI's o1 model on different tasks and summarizes six reasoning patterns of the o1 model (system analysis, method reuse, divide - and - conquer, self - correction, context recognition, and emphasis on constraints). These patterns help to understand how the o1 model performs well in complex tasks.

4. **Benchmark testing and data filtering**: In order to evaluate the performance of the model more accurately, the paper selects four benchmark test sets (HotpotQA, Collie, USACO, and AIME), covering three fields of common - sense reasoning, mathematics, and programming. In addition, a data filtering module is designed to exclude simple samples that can be easily answered by current LLMs, so as to better distinguish the performance differences between different models.

In summary, this paper aims to reveal the impact of test - time compute methods on the performance of LLMs through comparative experiments and detailed reasoning pattern analysis, and deeply explore the advantages and limitations of OpenAI's o1 model in complex tasks.