Abstract:Driven by recent advancements in Extended Reality (XR), the hype around the Metaverse, and real‐time computer graphics, the transformation of the performing arts, particularly in digitizing and visualizing musical experiences, is an ever‐evolving landscape. This transformation offers significant potential in promoting inclusivity, fostering creativity, and enabling live performances in diverse settings. However, despite its immense potential, the field of Virtual Instrument Performances (VIP) has remained relatively unexplored due to numerous challenges. These challenges arise from the complex and multi‐modal nature of musical instrument performances, the need for high precision motion capture under occlusions including the intricate interactions between a musician's body and fingers with instruments, the precise synchronization and seamless integration of various sensory modalities, accommodating variations in musicians' playing styles, facial expressions, and addressing instrument‐specific nuances. This comprehensive survey delves into the intersection of technology, innovation, and artistic expression in the domain of virtual instrument performances. It explores musical performance multi‐modal databases and investigates a wide range of data acquisition methods, encompassing diverse motion capture techniques, facial expression recording, and various approaches for capturing audio and MIDI data (Musical Instrument Digital Interface). The survey also explores Music Information Retrieval (MIR) tasks, with a particular emphasis on the Musical Performance Analysis (MPA) field, and offers an overview of various works in the realm of Musical Instrument Performance Synthesis (MIPS), encompassing recent advancements in generative models. The ultimate aim of this survey is to unveil the technological limitations, initiate a dialogue about the current challenges, and propose promising avenues for future research at the intersection of technology and the arts.

Musical Instruments in Extended Reality: A Systematic Review

Virtual Instrument Performances (VIP): A Comprehensive Review

Using Wearable Sensors to Study Musical Experience: A Systematic Review

Musical Metaverse Playgrounds: exploring the design of shared virtual sonic experiences on web browsers

The virtual drum circle: polyrhythmic music interactions in extended reality

Research Trends in Virtual Reality Music Concert Technology: A Systematic Literature Review

Research on human engagement with music in the Journal of New Music Research

A comprehensive review of sensors and instrumentation methods in devices for musical expression

Current status and clinical perspectives of extended reality for myoelectric prostheses: review

Extended reality for biomedicine

Augmented Reality Applications of Spanish Folk Music Elements in 7 Spanish Ballads and Don Quixote to the Intended Lovers

Musical Neurofeedback Advancements, Feedback Modalities, and Applications: A Systematic Review

Brain to Brain Musical Interaction: A Systematic Review of Neural Synchrony in Musical Activities

A Systematic Review of Extended Reality (XR) for Understanding and Augmenting Vision Loss

Audio-Tactile Rendering: A Review on Technology and Methods to Convey Musical Information through the Sense of Touch

A Comparison of Interfaces for Learning How to Play a Mixed Reality Handpan

Cyborg Children: A Systematic Literature Review on the Experience of Children Using Extended Reality

Designing an augmented experience for a music archive: What does the audience need beyond the sense of hearing?

Integration of music-based game approaches with wearable devices for hand neurorehabilitation: a narrative review

A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?

The user experience of mobile music making: An ethnographic exploration of music production and performance in practice