Abstract:With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user's facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user's face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users' facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user's facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user's eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user's facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.

Facial performance sensing head-mounted display

High-fidelity facial and speech animation for VR HMDs

Real-time 3D Face-Eye Performance Capture of a Person Wearing VR Headset

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction `

Facial Expressions Recognition Based on Convolutional Neural Networks for Mobile Virtual Reality.

Hands-Free User Interface for VR Headsets Based on In Situ Facial Gesture Sensing

HeadsetOff: Enabling Photorealistic Video Conferencing on Economical VR Headsets

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications

Universal Facial Encoding of Codec Avatars from VR Headsets

Attention-Based VR Facial Animation with Visual Mouth Camera Guidance for Immersive Telepresence Avatars

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral Cameras

Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Mm3dface: Nonintrusive 3D Facial Reconstruction Leveraging Mmwave Signals

VR Facial Animation for Immersive Telepresence Avatars

Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display

VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence

Head mounted device for gesture recognition using acoustic time of flight sensor and 2D camera