Abstract:Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.

What problem does this paper attempt to address?

The paper aims to address the issue of Full Duplex Modeling (FDM) in interactive voice dialogue systems, particularly enhancing real-time interaction capabilities and exploring the core function of interruption capability. Existing language models and speech language models (SLM) are typically limited to turn-taking dialogue modes, lacking the ability to interact with humans in real-time in real-world scenarios, such as being interrupted by humans when the generated content is unsatisfactory. To overcome these limitations, the researchers proposed a new model design called the "Listening-While-Speaking Language Model" (LSLM). LSLM is an end-to-end system that integrates both listening and speaking channels. It uses a token-based decoder for text-to-speech (TTS) technology to generate speech and employs a streaming self-supervised learning (SSL) encoder to process real-time audio input. LSLM merges these two channels for autoregressive generation and achieves real-time detection of dialogue turn-taking. The researchers also explored three fusion strategies—early fusion, mid-fusion, and late fusion, with mid-fusion achieving the best balance between speech generation and real-time interaction capabilities. Through two experimental setups—command-based full duplex modeling and speech-based full duplex modeling, the paper validated the robustness of LSLM in noisy environments and its sensitivity to different commands. The experimental results show that LSLM can achieve duplex communication with minimal negative impact on existing systems. The goal of this research is to advance the development of interactive voice dialogue systems and improve their usability in practical application scenarios.

Language Model Can Listen While Speaking

A Full-duplex Speech Dialogue Scheme Based On Large Language Models

Dialogue Learning with Human-in-the-Loop.

Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models

Enabling Real-Time Conversations with Minimal Training Costs

Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents

Towards Joint Modeling of Dialogue Response and Speech Synthesis based on Large Language Model

Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue

SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation

Large Language Model based Situational Dialogues for Second Language Learning

A Synthetic And Computational Language Model For Interactive Dialogue System

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Can Large Language Models Be Good Companions? An LLM-Based Eyewear System with Conversational Common Ground

Retrieval Augmented End-to-End Spoken Dialog Models

SLM: Bridge the thin gap between speech and text foundation models

SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems

Sibyl: Empowering Empathetic Dialogue Generation in Large Language Models via Sensible and Visionary Commonsense Inference

Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities

Recent Advances in Speech Language Models: A Survey