Drawlody: Sketch-Based Melody Creation with Enhanced Usability and Interpretability
Qihao Liang,Ye Wang
DOI: https://doi.org/10.1109/tmm.2024.3360695
IF: 7.3
2024-01-01
IEEE Transactions on Multimedia
Abstract:Sketch-based melody creation systems enable people to compose melodies by converting human-sketched melody contours into coherent melodies that fit the depicted contours. This remains one of the most intuitive approaches to interactive music creation. However, previous studies are still stagnating in limitations regarding usability and interpretability, which hinders effective interactions between people and AI. For one thing, these studies entail additional complex musical conditions as auxiliary inputs (e.g. chord progressions, contextual melodies, and predetermined rhythms), supporting only fixed-length and rule-based melody generation. This makes existing systems less usable, with generated melodies lacking diversity and coherence. Moreover, users without enough musical expertise might find it difficult to define appropriate inputs and to interpret the role of these inputs in guiding melody generation. To address these limitations, we present Drawlody, a novel sketch-based melody creation system with enhanced usability and interpretability. Specifically, Drawlody simplifies user input requirements by excluding all complex musical conditions, using only a simplified melody contour representation named Generalised Melody Contour (GMC) as input. This simplification clarifies the role of user controls, making the system more usable for people without musical training. To guide coherent melody generation from GMC, we propose FlexMIDI music representation, which simulates the tonal structure of melodies and faithfully explains how human-sketched contours guide melody generation. We employ a CNN-Transformer-based architecture as the foundation model to achieve arbitrary-length melody generation. Drawlody is evaluated by both objective and subjective music quality studies, as well as a usability and interpretability study. The results support its enhanced usability, interpretability, and high-quality melody generation capabilities. Video demos of the system are presented here.
computer science, information systems,telecommunications, software engineering