Abstract:The emerging conditional coding-based neural video codec (NVC) shows superiority over commonly-used residual coding-based codec and the latest NVC already claims to outperform the best traditional codec. However, there still exist critical problems blocking the practicality of NVC. In this paper, we propose a powerful conditional coding-based NVC that solves two critical problems via feature modulation. The first is how to support a wide quality range in a single model. Previous NVC with this capability only supports about 3.8 dB PSNR range on average. To tackle this limitation, we modulate the latent feature of the current frame via the learnable quantization scaler. During the training, we specially design the uniform quantization parameter sampling mechanism to improve the harmonization of encoding and quantization. This results in a better learning of the quantization scaler and helps our NVC support about 11.4 dB PSNR range. The second is how to make NVC still work under a long prediction chain. We expose that the previous SOTA NVC has an obvious quality degradation problem when using a large intra-period setting. To this end, we propose modulating the temporal feature with a periodically refreshing mechanism to boost the quality. %Besides solving the above two problems, we also design a single model that can support both RGB and YUV colorspaces. Notably, under single intra-frame setting, our codec can achieve 29.7\% bitrate saving over previous SOTA NVC with 16\% MACs reduction. Our codec serves as a notable landmark in the journey of NVC evolution. The codes are at

Neural Audio Coding with Deep Complex Networks

A High Fidelity and Low Complexity Neural Audio Coding

Neural Speech and Audio Coding

LightCodec: A High Fidelity Neural Audio Codec with Low Computation Complexity.

An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codec

SpatialCodec: Neural Spatial Speech Coding

Latent-Domain Predictive Neural Speech Coding

Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantization

Disentangled Feature Learning for Real-Time Neural Speech Coding

MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios

APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm

End-to-End Neural Speech Coding for Real-Time Communications

Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses

Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding

SNAC: Multi-Scale Neural Audio Codec

CQNV: A combination of coarsely quantized bitstream and neural vocoder for low rate speech coding

Neural Video Compression with Feature Modulation

Towards audio language modeling -- an overview

DeepCoder: A Deep Neural Network Based Video Compression

A High Efficient Audio Data Compression Scheme based on Neural Network