Abstract:In this paper, we touch on the problem of markerless multi-modal human motion capture especially for string performance capture which involves inherently subtle hand-string contacts and intricate movements. To fulfill this goal, we first collect a dataset, named String Performance Dataset (SPD), featuring cello and violin performances. The dataset includes videos captured from up to 23 different views, audio signals, and detailed 3D motion annotations of the body, hands, instrument, and bow. Moreover, to acquire the detailed motion annotations, we propose an audio-guided multi-modal motion capture framework that explicitly incorporates hand-string contacts detected from the audio signals for solving detailed hand poses. This framework serves as a baseline for string performance capture in a completely markerless manner without imposing any external devices on performers, eliminating the potential of introducing distortion in such delicate movements. We argue that the movements of performers, particularly the sound-producing gestures, contain subtle information often elusive to visual methods but can be inferred and retrieved from audio cues. Consequently, we refine the vision-based motion capture results through our innovative audio-guided approach, simultaneously clarifying the contact relationship between the performer and the instrument, as deduced from the audio. We validate the proposed framework and conduct ablation studies to demonstrate its efficacy. Our results outperform current state-of-the-art vision-based algorithms, underscoring the feasibility of augmenting visual motion capture with audio modality. To the best of our knowledge, SPD is the first dataset for musical instrument performance, covering fine-grained hand motion details in a multi-modal, large-scale collection. It holds significant implications and guidance for string instrument pedagogy, animation, and virtual concerts, as well as for both musical performance analysis and generation. Our code and SPD dataset are available at https://github.com/Yitongishere/string_performance.

Knowledge and data co-driven intelligent assessment of Chinese zither fingerings

A study on attention-based fine-grained image recognition: Towards musical instrument performing hand shape assessment

ChinAR: Facilitating Chinese Guqin Learning through Interactive Projected Augmentation.

At Your Fingertips: Extracting Piano Fingering Instructions from Videos

Innovative music education: Using an AI-based flipped classroom

Playing Technique Detection by Fusing Note Onset Information in Guzheng Performance

PianoWatch: an Intelligent Piano Understanding and Evaluation System Using Smartwatch

Study on Aesthetic Teaching Methods in Ethnic Music Teaching in Universities in the Context of Intelligent Internet of Things

Piano Fingering with Reinforcement Learning

PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

DrumGenius: bridging learning-gap with interactive musical instruments.

Challenges and Optimization Paths of Guzheng Professional Education in Colleges under Big Data Era

CCOM-HuQin: an Annotated Multimodal Chinese Fiddle Performance Dataset

Score Difficulty Analysis for Piano Performance Education based on Fingering

We Can Do More to Save Guqin: Design and Evaluate Interactive Systems to Make Guqin More Accessible to the General Public

Chinese Pre-service Music Teachers’ Perceptions of Augmented Reality-Assisted Musical Instrument Learning

Barehanded Music: Real-Time Hand Interaction For Virtual Piano

Design and preliminary evaluation of the FINGER rehabilitation robot: controlling challenge and quantifying finger individuation during musical computer game play

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

A Fuzzy Gestural AI Approach for Expressive Interactivity in Multitouch Digital Musical Instruments Based on Laban Descriptors

Musical Instrument Playing Technique Detection Based on FCN: Using Chinese Bowed-Stringed Instrument as an Example