Abstract:Piano playing requires agile, precise, and coordinated hand control that stretches the limits of dexterity. Hand motion models with the sophistication to accurately recreate piano playing have a wide range of applications in character animation, embodied AI, biomechanics, and VR/AR. In this paper, we construct a first-of-its-kind large-scale dataset that contains approximately 10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153 pieces of classical music. To capture natural performances, we designed a markerless setup in which motions are reconstructed from multi-view videos using state-of-the-art pose estimation models. The motion data is further refined via inverse kinematics using the high-resolution MIDI key-pressing data obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the collected dataset, we developed a pipeline that can synthesize physically-plausible hand motions for musical scores outside of the dataset. Our approach employs a combination of imitation learning and reinforcement learning to obtain policies for physics-based bimanual control involving the interaction between hands and piano keys. To solve the sampling efficiency problem with the large motion dataset, we use a diffusion model to generate natural reference motions, which provide high-level trajectory and fingering (finger order and placement) information. However, the generated reference motion alone does not provide sufficient accuracy for piano performance modeling. We then further augmented the data by using musical similarity to retrieve similar motions from the captured dataset to boost the precision of the RL policy. With the proposed method, our model generates natural, dexterous motions that generalize to music from outside the training dataset.

Piano Fingering with Reinforcement Learning

Towards Learning to Play Piano with Dexterous Hands and Touch

At Your Fingertips: Extracting Piano Fingering Instructions from Videos

RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning

FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance

Estimation of Playable Piano Fingering by Pitch-difference Fingering Matching Model

Score Difficulty Analysis for Piano Performance Education based on Fingering

Exploring Dual Dimensions in Piano Performance Practice Based on Modern Digital Technology

PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

Play it by Ear: A perceptual algorithm for autonomous melodious piano playing with a bio-inspired robotic hand

Piano Genie

Transfer Learning for Piano Sustain-Pedal Detection

RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands

Barehanded Music: Real-Time Hand Interaction For Virtual Piano

Application and Research of Machine Learning Algorithms in Personalized Piano Teaching System

Applying Deep Learning Techniques to Estimate Patterns of Musical Gesture

PianoWatch: an Intelligent Piano Understanding and Evaluation System Using Smartwatch

Hierarchical Optimal Motion Planning for Piano Playing Robots With Dexterous Fingers

Machine Learning for a Music Glove Instrument

On the Feasibility of Learning Finger-gaiting In-hand Manipulation with Intrinsic Sensing

Feeling the beat: a smart hand exoskeleton for learning to play musical instruments