Abstract:Recent advancements in test time compute, particularly through the use of verifier models, have significantly enhanced the reasoning capabilities of Large Language Models (LLMs). This generator-verifier approach closely resembles the actor-critic framework in reinforcement learning (RL). However, current verifier models in LLMs often rely on supervised fine-tuning without temporal difference learning such as Q-learning. This paper introduces VerifierQ, a novel approach that integrates Offline Q-learning into LLM verifier models. We address three key challenges in applying Q-learning to LLMs: (1) handling utterance-level Markov Decision Processes (MDPs), (2) managing large action spaces, and (3) mitigating overestimation bias. VerifierQ introduces a modified Bellman update for bounded Q-values, incorporates Implicit Q-learning (IQL) for efficient action space management, and integrates a novel Conservative Q-learning (CQL) formulation for balanced Q-value estimation. Our method enables parallel Q-value computation and improving training efficiency. While recent work has explored RL techniques like MCTS for generators, VerifierQ is among the first to investigate the verifier (critic) aspect in LLMs through Q-learning. This integration of RL principles into verifier models complements existing advancements in generator techniques, potentially enabling more robust and adaptive reasoning in LLMs. Experimental results on mathematical reasoning tasks demonstrate VerifierQ's superior performance compared to traditional supervised fine-tuning approaches, with improvements in efficiency, accuracy and robustness. By enhancing the synergy between generation and evaluation capabilities, VerifierQ contributes to the ongoing evolution of AI systems in addressing complex cognitive tasks across various domains.

Improving task independent utterance verification based on on-line garbage phoneme likelihood

Task Independent Utterance Verification Using Garbage Model Based On Hierarchical Phoneme

Enhancing Out-of-Domain Detection for Speech Spoofing Countermeasure Via Supervised Contrastive Learning

Experimental Investigation into Alignment-based Acoustic Confidence Measures in Keyword Verification for Mandarin Speech

Toward Pitch-Insensitive Speaker Verification Via Soundfield

A New High-Efficient Language Verification System

Utterance verification based on dynamic garbage evaluation approach

An utterance verification algorithm in keyword spotting system

Using Word Confidence Measure for OOV Words Detection in a Spontaneous Spoken Dialog System.

Combining Lattice-Based Language Dependent and Independent Approaches for Out-of-Language Detection in LVCSR.

Confidence Scoring Using Context Dependent Features for Word Verification

An lstm-ctc based verification system for proxy-word based oov keyword search

Improving Speaker Verification Performance Against Long-Term Speaker Variability

Addressing the Out-of-vocabulary Problem for Large-Scale Chinese Spoken Term Detection

Robust Training for Speaker Verification Against Noisy Labels

Oriental Language Recognition (OLR) 2020: Summary and Analysis

Improving Short Utterance PLDA Speaker Verification using SUV Modelling and Utterance Partitioning Approach

VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers

Investigation of Frame Alignments for GMM-based Digit-prompted Speaker Verification.

Utterance verification using modified segmental probability model

Speech Verification Based On Simplified Segmental Probability Model