Abstract:The fundamental frequency (F0) of human voice is generally controlled by changing the vocal fold parameters (including tension, length and mass), which in turn is manipulated by the muscle exciters, activated by the neural synergies. In order to begin investigating the neuromuscular F0 control pathway, we simulate a simple biomechanical arm prototype (instead of an artificial vocal tract) that tends to control F0 of an artificial sound synthesiser based on the elbow movements. The intended arm movements are decoded from the EEG signal inputs (collected simultaneously with the kinematic hand data of the participant) through a combined machine learning and biomechanical modeling strategy. The machine learning model is employed to identify the muscle activation of a single-muscle arm model in ArtiSynth (from input brain signal), in order to match the actual kinematic (elbow joint angle) data . The biomechanical model utilises this estimated muscle excitation to produce corresponding changes in elbow angle, which is then linearly mapped to F0 of a vocal sound synthesiser. We use the F0 value mapped from the actual kinematic hand data (via same function) as the ground truth and compare the F0 estimated from brain signal. A detailed qualitative and quantitative performance comparison shows that the proposed neuromuscular pathway can indeed be utilised to accurately control the vocal fundamental frequency, thereby demonstrating the success of our closed loop neuro-biomechanical control scheme.

Sound-Stream II: Towards Real-Time Gesture Controlled Articulatory Sound Synthesis

SPEAK WITH YOUR HANDS Using Continuous Hand Gestures to control Articulatory Speech Synthesizer

Artimate: an articulatory animation framework for audiovisual speech synthesis

Hearing how you touch: Real-time synthesis of contact sounds for multisensory interaction

Intuitive Control of Scraping and Rubbing Through Audio-tactile Synthesis

Towards Streaming Speech-to-Avatar Synthesis

Coding Speech through Vocal Tract Kinematics

Improvements of a Physiological Articulatory Model in Construction and Control Strategy

SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration

The Use of Articulatory Movement Data in Speech Synthesis Applications: an Overview — Application of Articulatory Movements Using Machine Learning Algorithms —

Embodying Spatial Sound Synthesis with AI in Two Compositions for Instruments and 3-D Electronics

Real-time Contact Sound Synthesis for Multisensory Interaction

EEG-to-F0: Establishing artificial neuro-muscular pathway for kinematics-based fundamental frequency control

Primate Drum Kit: A System for Studying Acoustic Pattern Production by Non-Human Primates Using Acceleration and Strain Sensors

A Novel Face-tracking Mouth Controller and its Application to Interacting with Bioacoustic Models

Toward Inverse Control of Physics-Based Sound Synthesis

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

Progress in animation of an EMA-controlled tongue model for acoustic-visual speech synthesis

Speech synthesis using a physiological articulatory model with feature-based rules

Designing, Playing, and Performing with a Vision-based Mouth Interface

MOONMENT: Designing Gesture-based Interaction with Acousto-Optic Feedbacks