Abstract:Recent advancements in audio generation have been significantly propelled by the capabilities of Large Language Models (LLMs). The existing research on audio LLM has primarily focused on enhancing the architecture and scale of audio language models, as well as leveraging larger datasets, and generally, acoustic codecs, such as EnCodec, are used for audio tokenization. However, these codecs were originally designed for audio compression, which may lead to suboptimal performance in the context of audio LLM. Our research aims to address the shortcomings of current audio LLM codecs, particularly their challenges in maintaining semantic integrity in generated audio. For instance, existing methods like VALL-E, which condition acoustic token generation on text transcriptions, often suffer from content inaccuracies and elevated word error rates (WER) due to semantic misinterpretations of acoustic tokens, resulting in word skipping and errors. To overcome these issues, we propose a straightforward yet effective approach called X-Codec. X-Codec incorporates semantic features from a pre-trained semantic encoder before the Residual Vector Quantization (RVQ) stage and introduces a semantic reconstruction loss after RVQ. By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation. Our experiments in text-to-speech, music continuation, and text-to-sound tasks demonstrate that integrating semantic information substantially improves the overall performance of language models in audio generation. Our code and demo are available (Demo: <a class="link-external link-https" href="https://x-codec-audio.github.io" rel="external noopener nofollow">this https URL</a> Code: <a class="link-external link-https" href="https://github.com/zhenye234/xcodec" rel="external noopener nofollow">this https URL</a>)

SemanticAC: Semantics-Assisted Framework for Audio Classification

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

A Hybrid Approach to Acoustic Scene Classification Based on Universal Acoustic Models.

Deep semantic learning for acoustic scene classification

Introducing Semantics into Speech Encoders.

Advanced Framework for Animal Sound Classification With Features Optimization

Robust Audio Sensing with Multi-Sound Classification.

Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information

Semantic Concept Annotation Based on Audio PLSA Model

Can Textual Semantics Mitigate Sounding Object Segmentation Preference?

ACES: Evaluating Automated Audio Captioning Models on the Semantics of Sounds

Deep Segment Model for Acoustic Scene Classification

An Acoustic Segment Model Based Segment Unit Selection Approach to Acoustic Scene Classification with Partial Utterances

Synth-AC: Enhancing Audio Captioning with Synthetic Supervision

Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations

Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

Improving Acoustic Scene Classification Via Self-Supervised and Semi-Supervised Learning with Efficient Audio Transformer

Few-shot Class-incremental Audio Classification Using Dynamically Expanded Classifier with Self-attention Modified Prototypes

Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics