Abstract:Despite the significant progress in controllable music generation and editing, challenges remain in the quality and length of generated music due to the use of Mel-spectrogram representations and UNet-based model structures. To address these limitations, we propose a novel approach using a Diffusion Transformer (DiT) augmented with an additional control branch using ControlNet. This allows for long-form and variable-length music generation and editing controlled by text and melody prompts. For more precise and fine-grained melody control, we introduce a novel top-$k$ constant-Q Transform representation as the melody prompt, reducing ambiguity compared to previous representations (e.g., chroma), particularly for music with multiple tracks or a wide range of pitch values. To effectively balance the control signals from text and melody prompts, we adopt a curriculum learning strategy that progressively masks the melody prompt, resulting in a more stable training process. Experiments have been performed on text-to-music generation and music-style transfer tasks using open-source instrumental recording data. The results demonstrate that by extending StableAudio, a pre-trained text-controlled DiT model, our approach enables superior melody-controlled editing while retaining good text-to-music generation performance. These results outperform a strong MusicGen baseline in terms of both text-based generation and melody preservation for editing. Audio examples can be found at <a class="link-external link-https" href="https://stable-audio-control.github.io/web/" rel="external noopener nofollow">this https URL</a>.

METEOR: Melody-aware Texture-controllable Symbolic Orchestral Music Generation

SongDriver2: Real-time Emotion-based Music Arrangement with Soft Transition.

Flexible Control in Symbolic Music Generation via Musical Metadata

Crafting Creative Melodies: A User-Centric Approach for Symbolic Music Generation

A Multi-Scale Attentive Transformer for Multi-Instrument Symbolic Music Generation

Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset

Emotion-Driven Melody Harmonization via Melodic Variation and Functional Representation

MeloTrans: A Text to Symbolic Music Generation Model Following Human Composition Habit

Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer

Symbolic Music Generation with Fine-grained Interactive Textural Guidance

Symphony Generation with Permutation Invariant Language Model

Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer

Practical and Reproducible Symbolic Music Generation by Large Language Models with Structural Embeddings

Accompanied Singing Voice Synthesis with Fully Text-controlled Melody

Melody-Guided Music Generation

MusicAOG: an Energy-Based Model for Learning and Sampling a Hierarchical Representation of Symbolic Music

Melody Structure Transfer Network: Generating Music with Separable Self-Attention

MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks

Hierarchical Timbre-Painting and Articulation Generation

Symbolic Music Playing Techniques Generation as a Tagging Problem

Text2midi: Generating Symbolic Music from Captions