Abstract:Emotion is one of the most crucial attributes of music. However, due to the scarcity of emotional music datasets, emotion-conditioned symbolic music generation using deep learning techniques has not been investigated in depth. In particular, no study explores conditional music generation with the guidance of emotion, and few studies adopt time-varying emotional conditions. To address these issues, first, we endow three public lead sheet datasets with fine-grained emotions by automatically computing the valence labels from the chord progressions. Second, we propose a novel and effective encoder-decoder architecture named EmoMusicTV to explore the impact of emotional conditions on multiple music generation tasks and to capture the rich variability of musical sequences. EmoMusicTV is a transformer-based variational autoencoder (VAE) that contains a hierarchical latent variable structure to model holistic properties of the music segments and short-term variations within bars. The piece-level and bar-level emotional labels are embedded in their corresponding latent spaces to guide music generation. Third, we pretrain EmoMusicTV with the lead sheet continuation task to further improve its performance on conditional melody or harmony generation. Experimental results demonstrate that EmoMusicTV outperforms previous methods on three tasks, i.e., melody harmonization, melody generation given harmony, and lead sheet generation. Ablation studies verify the significant roles of emotional conditions and hierarchical latent variable structure on conditional music generation. Human listening shows that the lead sheets generated by EmoMusicTV are closer to the ground truth (GT) and perform slightly worse than the GT in conveying emotional polarity.

Application and Research of Music Generation System Based on CVAE and Transformer-XL in Video Background Music

A Sentiment-Controllable Music Generation System Based on Conditional Variational Autoencoder

Double Linear Transformer for Background Music Generation from Videos

An Intelligent Music Generation Based on Variational Autoencoder

Video Background Music Generation with Controllable Music Transformer

Music Generation System for Adversarial Training Based on Deep Learning

VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos

EmoMusicTV: Emotion-Conditioned Symbolic Music Generation with Hierarchical Transformer VAE

Transformer-XL Based Music Generation with Multiple Sequences of Time-valued Notes

Application Research of Short-Time Fourier Transform in Music Generation Based on the Parallel WaveGan System

A Music-Driven System for Generating Apparel Display Video

Video Echoed in Harmony: Learning and Sampling Video-Integrated Chord Progression Sequences for Controllable Video Background Music Generation

A System For Automatic Generation Of Music Sports-Video

MAGMA: Music Aligned Generative Motion Autodecoder

Music Generation based on Generative Adversarial Networks with Transformer

Application and Research of Monte Carlo Sampling Algorithm in Music Generation

Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model

Music Conditioned Generation for Human-Centric Video

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Emotion-Guided Image to Music Generation

A Novel Machine Learning Algorithm: Music Arrangement and Timbre Transfer System