Abstract:Background: Virtual humans have become part of our everyday life (movies, internet, and computer games). Even though they are becoming more and more realistic, their speech capabilities are, most of the time, limited and not coherent and/or not synchronous with the corresponding acoustic signal. Methods: We describe a method to convert a virtual human avatar (animated through key frames and interpolation) into a more naturalistic talking head. In fact, speech articulation cannot be accurately replicated using interpolation between key frames and talking heads with good speech capabilities are derived from real speech production data. Motion capture data are commonly used to provide accurate facial motion for visible speech articulators (jaw and lips) synchronous with acoustics. To access tongue trajectories (partially occluded speech articulator), electromagnetic articulography (EMA) is often used. We recorded a large database of phonetically-balanced English sentences with synchronous EMA, motion capture data, and acoustics. An articulatory model was computed on this database to recover missing data and to provide 'normalized' animation (i.e., articulatory) parameters. In addition, semi-automatic segmentation was performed on the acoustic stream. A dictionary of multimodal Australian English diphones was created. It is composed of the variation of the articulatory parameters between all the successive stable allophones. Results: The avatar's facial key frames were converted into articulatory parameters steering its speech articulators (jaw, lips and tongue). The speech production database was used to drive the Embodied Conversational Agent (ECA) and to enhance its speech capabilities. A Text-To-Auditory Visual Speech synthesizer was created based on the MaryTTS software and on the diphone dictionary derived from the speech production database. Conclusions: We describe a method to transform an ECA with generic tongue model and animation by key frames into a talking head that displays naturalistic tongue, jaw and lip motions. Thanks to a multimodal speech production database, a Text-To-Auditory Visual Speech synthesizer drives the ECA's facial movements enhancing its speech capabilities.

Speech technology and cinema: Can they learn from each other?

Emotional Storytelling Using Virtual and Robotic Agents

Semiotic Analysis of Robot Sounds in Films: Implications for Sound Design in Social Robotics

Voice Synthesis Improvement by Machine Learning of Natural Prosody

Exploring expressivity and emotion with artificial voice and speech technologies

Giving Robots a Voice: Human-in-the-Loop Voice Creation and open-ended Labeling

Read the Room: Adapting a Robot's Voice to Ambient and Social Contexts

Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans

Digital Ventriloquism: Giving Voice to Everyday Objects

Transforming an embodied conversational agent into an efficient talking head: from keyframe-based animation to multimodal concatenation synthesis

Vocal Aesthetics, AI Imaginaries

No More Mumbles: Enhancing Robot Intelligibility through Speech Adaptation

Suggestive verbalizations in film: on character speech and sensory imagination

The Role of Artificial Intelligence in Voice-Based Systems: A Comparative Analysis of SAARUSH

Whither the Priors for (Vocal) Interactivity?

Fiction meets fact: exploring human-machine convergence in today’s cinematographic culture

Building for Speech: Designing the Next Generation of Social Robots for Audio Interaction

Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis

An Initial Exploration: Learning to Generate Realistic Audio for Silent Video

A Comprehensive Review and Taxonomy of Audio-Visual Synchronization Techniques for Realistic Speech Animation

Character-aware audio-visual subtitling in context