Abstract:Most existing neural network models for music generation explore how to generate music bars, then directly splice the music bars into a song. However, these methods do not explore the relationship between the bars, and the connected song as a whole has no musical form structure and sense of musical direction. To address this issue, we propose a Multi-model Multi-task Hierarchical Conditional VAE-GAN (Variational Autoencoder-Generative adversarial networks) networks, named MIDI-Sandwich, which combines musical knowledge, such as musical form, tonic, and melodic motion. The MIDI-Sandwich has two submodels: Hierarchical Conditional Variational Autoencoder (HCVAE) and Hierarchical Conditional Generative Adversarial Network (HCGAN). The HCVAE uses hierarchical structure. The underlying layer of HCVAE uses Local Conditional Variational Autoencoder (L-CVAE) to generate a music bar which is pre-specified by the First and Last Notes (FLN). The upper layer of HCVAE uses Global Variational Autoencoder(G-VAE) to analyze the latent vector sequence generated by the L-CVAE encoder, to explore the musical relationship between the bars, and to produce the song pieced together by multiple music bars generated by the L-CVAE decoder, which makes the song both have musical structure and sense of direction. At the same time, the HCVAE shares a part of itself with the HCGAN to further improve the performance of the generated music. The MIDI-Sandwich is validated on the Nottingham dataset and is able to generate a single-track melody sequence (17x8 beats), which is superior to the length of most of the generated models (8 to 32 beats). Meanwhile, by referring to the experimental methods of many classical kinds of literature, the quality evaluation of the generated music is performed. The above experiments prove the validity of the model.

Network Modulation Synthesis: New Algorithms for Generating Musical Audio Using Autoencoder Networks

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

GANSynth: Adversarial Neural Audio Synthesis

NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation

SynthNet: Learning to Synthesize Music End-to-End

Creative Text-to-Audio Generation via Synthesizer Programming

Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Towards Audio to Scene Image Synthesis using Generative Adversarial Network

SING: Symbol-to-Instrument Neural Generator

Analysis by Adversarial Synthesis -- A Novel Approach for Speech Vocoding

NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks

MusicHiFi: Fast High-Fidelity Stereo Vocoding

Conditioning Deep Generative Raw Audio Models for Structured Automatic Music

MIDI-Sandwich: Multi-model Multi-task Hierarchical Conditional VAE-GAN networks for Symbolic Single-track Music Generation

PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models

Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?

Deep generative models for musical audio synthesis

MelNet: A Generative Model for Audio in the Frequency Domain

VampNet: Music Generation via Masked Acoustic Token Modeling

A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis