Abstract:Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we request the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct the wrong reasoning after the RL stage. The RL procedure requires massive efforts for hyperparameter tuning and often generates errors like repetitive words and incomplete sentences. With the correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on multi-model dataset ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. ARES rationale reasoning achieves around 70% win rate against baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.

What problem does this paper attempt to address?

The paper attempts to address two main issues: 1. **Limitations of existing reinforcement learning methods (such as Reinforcement Learning from Human Feedback, RLHF, and Reinforcement Learning from AI Feedback, RLAIF)**: - These methods typically use rank-based feedback to optimize generated content, but this approach may overlook certain valuable segments, causing the model to fail to fully utilize this information. - The reinforcement learning process itself is unstable and requires extensive hyperparameter tuning, which becomes particularly challenging as the model scales up, leading to repetitive words and incomplete sentences. 2. **Improving the quality of Chain-of-Thought (CoT) reasoning in multimodal tasks**: - Although existing multimodal models perform well across various tasks, they still have shortcomings in generating chain-of-thought reasoning, especially when dealing with complex multimodal data. - The paper proposes a new two-stage algorithm, ARES (Alternating Reinforcement Learning and Supervised Fine-Tuning), which enhances chain-of-thought reasoning in multimodal tasks through diverse feedback provided by advanced AI models (such as GPT-4 and Claude 3 Opus). Specifically, the ARES algorithm addresses the above issues through the following steps: 1. **Reinforcement Learning (RL) Stage**: - Advanced AI models score each sentence on a scale from 0.0 to 1.0, indicating the sentence's contribution to solving the problem. - These scores are used as rewards in reinforcement learning, allowing the model to optimize the generated content more precisely and improve the quality of chain-of-thought reasoning. 2. **Supervised Fine-Tuning (SFT) Stage**: - After the reinforcement learning stage, advanced AI models correct the generated content, particularly targeting errors such as repetitive sentences and incomplete sentences. - The corrected data is then used for supervised fine-tuning, further stabilizing the model and enhancing its performance in multimodal tasks. Through these steps, the ARES algorithm not only improves the quality of chain-of-thought reasoning but also significantly enhances the accuracy of reasoning answers in multimodal tasks. Experimental results show that ARES outperforms baseline models across multiple datasets.

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback

ReFT: Reasoning with Reinforced Fine-Tuning

Making Large Language Models Better Reasoners with Alignment

Teaching Large Language Models to Reason with Reinforcement Learning

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

A Critical Evaluation of AI Feedback for Aligning Large Language Models

MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning

ReAct: Synergizing Reasoning and Acting in Language Models

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models

REFINER: Reasoning Feedback on Intermediate Representations

SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

Fine-tuning Language Models with Generative Adversarial Feedback

OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrieval

A NotSo Simple Way to Beat Simple Bench

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback