Abstract:Neural speech codecs have recently emerged as a focal point in the fields of speech compression and generation. Despite this progress, achieving high-quality speech reconstruction under low-bitrate scenarios remains a significant challenge. In this paper, we propose PSCodec, a series of neural speech codecs based on prompt encoders, comprising PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN, which are capable of delivering high-performance speech reconstruction with low bandwidths. Specifically, we first introduce PSCodec-Base, which leverages a pretrained speaker verification model-based prompt encoder (VPP-Enc) and a learnable Mel-spectrogram-based prompt encoder (MelP-Enc) to effectively disentangle and integrate voiceprint and Mel-related features in utterances. To further enhance feature utilization efficiency, we propose PSCodec-DRL-ICT, incorporating a structural similarity (SSIM) based disentangled representation loss (DRL) and an incremental continuous training (ICT) strategy. While PSCodec-DRL-ICT demonstrates impressive performance, its reliance on extensive hyperparameter tuning and multi-stage training makes it somewhat labor-intensive. To circumvent these limitations, we propose PSCodec-CasAN, utilizing an advanced cascaded attention network (CasAN) to enhance representational capacity of the entire system. Extensive experiments show that our proposed PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN all significantly outperform several state-of-the-art neural codecs, exhibiting substantial improvements in both speech reconstruction quality and speaker similarity under low-bitrate conditions.

TS3-Codec: Transformer-Based Simple Streaming Single Codec

Ultra-Low-Bitrate Speech Coding with Pretrained Transformers

MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios

AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec

Towards Audio Codec-based Speech Separation

SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

SSNVC: Single Stream Neural Video Compression with Implicit Temporal Information

ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

SoundStream: An End-to-End Neural Audio Codec

SNAC: Multi-Scale Neural Audio Codec

Neural Audio Coding with Deep Complex Networks

Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding

FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec

A High Fidelity and Low Complexity Neural Audio Coding

PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders

FreeCodec: A disentangled neural speech codec with fewer tokens

Latent-Domain Predictive Neural Speech Coding

C3: High-performance and low-complexity neural compression from a single image or video

Standard compliant video coding using low complexity, switchable neural wrappers

SpatialCodec: Neural Spatial Speech Coding

STC: Significance-aware Transform-based Codec Framework for External Memory Access Reduction