Abstract:Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the pianorolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between pianorolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two off-the-shelf synthesizers. We open our source code at https://github.com/bwang514/PerformanceNet

A holistic approach to polyphonic music transcription with neural networks

Polyphonic Piano Transcription Based on Graph Convolutional Network

Polyphonic pitch detection with convolutional recurrent neural networks

A Parallel Fusion Approach to Piano Music Transcription Based on Convolutional Neural Network

Multimodal image and audio music transcription

End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding

Multi-Instrument Polyphonic Melody Transcription Based on Deep Learning

Improved Architecture for High-resolution Piano Transcription to Efficiently Capture Acoustic Characteristics of Music Signals

End-to-end Piano Performance-MIDI to Score Conversion with Transformers

DAFE-MSGAT: Dual-Attention Feature Extraction and Multi-Scale Graph Attention Network for Polyphonic Piano Transcription

PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Towards Efficient and Real-Time Piano Transcription Using Neural Autoregressive Models

Scorpiano -- A System for Automatic Music Transcription for Monophonic Piano Music

Coordinate Embedding Transformer Model for Optical Music Recognition on Monophonic Scores

Polyphonic Music Composition with LSTM Neural Networks and Reinforcement Learning

From Music Scores to Audio Recordings: Deep Pitch-Class Representations for Measuring Tonal Structures

A multimodal approach to music transcription

Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription

Polyphonic Music Modelling with LSTM-RTRBM

Stereo Feature Enhancement and Temporal Information Extraction Network for Automatic Music Transcription

Aligned Music Notation and Lyrics Transcription