Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

Xin Lai,Zhuotao Tian,Yukang Chen,Senqiao Yang,Xiangru Peng,Jiaya Jia
2024-06-27
Abstract:Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by learning from human feedback. However, Direct Preference Optimization (DPO) has shown limited benefits for long-chain mathematical reasoning, as models employing DPO struggle to identify detailed errors in incorrect answers. This limitation stems from a lack of fine-grained process supervision. We propose a simple, effective, and data-efficient method called Step-DPO, which treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Additionally, we have developed a data construction pipeline for Step-DPO, enabling the creation of a high-quality dataset containing 10K step-wise preference pairs. We also observe that in DPO, self-generated data is more effective than data generated by humans or GPT-4, due to the latter's out-of-distribution nature. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at <a class="link-external link-https" href="https://github.com/dvlab-research/Step-DPO" rel="external noopener nofollow">this https URL</a>.
Machine Learning,Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
The paper mainly investigates the limitations of large language models (LLMs) in mathematical reasoning, particularly the challenges posed by the precision and length requirements of the reasoning chain. Current methods face difficulties in ensuring the correctness of each step of reasoning. To address this issue, the paper proposes "Step-DPO" (Step-Wise Preference Optimization), which optimizes preferences on a step-by-step basis rather than evaluating the answer as a whole. Traditional direct preference optimization (DPO) performs poorly in long-chain mathematical reasoning as it struggles to identify the details of incorrect answers. The paper introduces a data construction process for generating high-quality datasets suitable for Step-DPO, consisting of approximately 10,000 step-wise preference pairs. The study finds that using self-generated data from the model is more effective than data generated by humans or GPT-4. Through Step-DPO training, the model achieves an accuracy improvement of nearly 3% on the MATH test set, showcasing significant performance gains for a model with over 70 billion parameters, surpassing multiple closed-source code models including GPT-4. Experimental results demonstrate that Step-DPO significantly enhances mathematical reasoning abilities, especially for problems requiring multi-step thinking. Through comparative experiments, the paper proves the superiority of Step-DPO over direct preference optimization and showcases its effectiveness across different model scales. Additionally, the paper explores the differences with other reinforcement learning methods, emphasizing the superiority of Step-DPO in long-chain reasoning tasks.