Abstract:Neural speech codecs have recently emerged as a focal point in the fields of speech compression and generation. Despite this progress, achieving high-quality speech reconstruction under low-bitrate scenarios remains a significant challenge. In this paper, we propose PSCodec, a series of neural speech codecs based on prompt encoders, comprising PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN, which are capable of delivering high-performance speech reconstruction with low bandwidths. Specifically, we first introduce PSCodec-Base, which leverages a pretrained speaker verification model-based prompt encoder (VPP-Enc) and a learnable Mel-spectrogram-based prompt encoder (MelP-Enc) to effectively disentangle and integrate voiceprint and Mel-related features in utterances. To further enhance feature utilization efficiency, we propose PSCodec-DRL-ICT, incorporating a structural similarity (SSIM) based disentangled representation loss (DRL) and an incremental continuous training (ICT) strategy. While PSCodec-DRL-ICT demonstrates impressive performance, its reliance on extensive hyperparameter tuning and multi-stage training makes it somewhat labor-intensive. To circumvent these limitations, we propose PSCodec-CasAN, utilizing an advanced cascaded attention network (CasAN) to enhance representational capacity of the entire system. Extensive experiments show that our proposed PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN all significantly outperform several state-of-the-art neural codecs, exhibiting substantial improvements in both speech reconstruction quality and speaker similarity under low-bitrate conditions.

Variable-rate Neural Speech Compression with Multi-scale Feature Extraction and Improved Entropy Modeling

Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding

Variational Speech Waveform Compression to Catalyze Semantic Communications

Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantization

ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

CQNV: A combination of coarsely quantized bitstream and neural vocoder for low rate speech coding

Rate–Distortion–Perception Optimized Neural Speech Transmission System for High-Fidelity Semantic Communications

One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model

Srcodec: Split-Residual Vector Quantization for Neural Speech Codec.

An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codec

FreeCodec: A disentangled neural speech codec with fewer tokens

VRVQ: Variable Bitrate Residual Vector Quantization for Audio Compression

Practical cognitive speech compression

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs

Ultra-Low-Bitrate Speech Coding with Pretrained Transformers

PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders

Neural Audio Coding with Deep Complex Networks

QVRF: A Quantization-error-aware Variable Rate Framework for Learned Image Compression

A High Fidelity and Low Complexity Neural Audio Coding