Abstract:The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine--musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.

Notochord: a Flexible Probabilistic Model for Real-Time MIDI Performance

An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks

Understanding Musical Predictions With an Embodied Interface for Musical Machine Learning

Real-time jam-session support system

Deep Predictive Models in Interactive Music

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System

Striking a New Chord: Neural Networks in Music Information Dynamics

Local deployment of large-scale music AI models on commodity hardware

Live Orchestral Piano, a system for real-time orchestral music generation

PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations

JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs

Flat Latent Manifolds for Human-machine Co-creation of Music

MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

JS Fake Chorales: a Synthetic Dataset of Polyphonic Music with Human Annotation

Anticipatory Music Transformer

NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting

Human-Robot Cooperative Piano Playing with Learning-Based Real-Time Music Accompaniment

Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation

Performing Structured Improvisations with pre-trained Deep Learning Models