Partin Tables cannot accurately predict the pathological stage at radical prostatectomy.

N. Bhojani,S. Ahyai,M. Graefen,U. Capitanio,N. Suardi,S. Shariat,C. Jeldres,A. Erbersdobler,T. Schlomm,A. Haese,T. Steuber,H. Heinzer,F. Montorsi,Hartwig Huland,P. Karakiewicz

DOI: https://doi.org/10.1016/j.ejso.2008.07.013

IF: 4.037

2009-02-01

European Journal of Surgical Oncology

Abstract:

What problem does this paper attempt to address?

Stereo Camera System and Its Application in Networked Virtual Environment

L Chen,MH Tian,GC Chen,C Chen

DOI: https://doi.org/10.1109/icmlc.2002.1176744

2002-01-01

Abstract:With the appearance of different kinds of networked virtual environment systems, the technology of realistic video avatar has become a hotspot of research. How to create realistic video avatar with less hardware investment and limited computation is one of the purposes of these researches. We introduce a method that combines a single stereo camera system and a 3D head model to create a realistic video avatar. To create the video avatar's most important part-the face, we use a 2D image and a depth map, which is generated by the stereo camera system. The other parts of of the video avatar are created by mapping the head's texture to the 3D head model.
Mimetic Muscle Rehabilitation Analysis Using Clustering of Low Dimensional 3D Kinect Data

Sumit Kumar Vishwakarma,Sanjeev Kumar,Shrey Aggarwal,Jan Mareš

DOI: https://doi.org/10.48550/arXiv.2302.09295

2023-02-15

Computers and Society

Abstract:Facial nerve paresis is a severe complication that arises post-head and neck surgery; This results in articulation problems, facial asymmetry, and severe problems in non-verbal communication. To overcome the side effects of post-surgery facial paralysis, rehabilitation requires which last for several weeks. This paper discusses an unsupervised approach to rehabilitating patients who have temporary facial paralysis due to damage in mimetic muscles. The work aims to make the rehabilitation process objective compared to the current subjective approach, such as House-Brackmann (HB) scale. Also, the approach will assist clinicians by reducing their workload in assessing the improvement during rehabilitation. This paper focuses on the clustering approach to monitor the rehabilitation process. We compare the results obtained from different clustering algorithms on various forms of the same data set, namely dynamic form, data expressed as functional data using B-spline basis expansion, and by finding the functional principal components of the functional data. The study contains data set of 85 distinct patients with 120 measurements obtained using a Kinect stereo-vision camera. The method distinguish effectively between patients with the least and greatest degree of facial paralysis, however patients with adjacent degrees of paralysis provide some challenges. In addition, we compared the cluster results to the HB scale outputs.
Novel Baseline Facial Muscle Database Using Statistical Shape Modeling and In Silico Trials toward Decision Support for Facial Rehabilitation

Vi-Do Tran,Tan-Nhu Nguyen,Abbass Ballit,Tien-Tuan Dao

DOI: https://doi.org/10.3390/bioengineering10060737

IF: 5.046

2023-06-19

Bioengineering

Abstract:Backgrounds and Objective: Facial palsy is a complex pathophysiological condition affecting the personal and professional lives of the involved patients. Sudden muscle weakness or paralysis needs to be rehabilitated to recover a symmetric and expressive face. Computer-aided decision support systems for facial rehabilitation have been developed. However, there is a lack of facial muscle baseline data to evaluate the patient states and guide as well as optimize the rehabilitation strategy. In this present study, we aimed to develop a novel baseline facial muscle database (static and dynamic behaviors) using the coupling between statistical shape modeling and in-silico trial approaches. Methods: 10,000 virtual subjects (5000 males and 5000 females) were generated from a statistical shape modeling (SSM) head model. Skull and muscle networks were defined so that they statistically fit with the head shapes. Two standard mimics: smiling and kissing were generated. The muscle strains of the lengths in neutral and mimic positions were computed and recorded thanks to the muscle insertion and attachment points on the animated head and skull meshes. For validation, five head and skull meshes were reconstructed from the five computed tomography (CT) image sets. Skull and muscle networks were then predicted from the reconstructed head meshes. The predicted skull meshes were compared with the reconstructed skull meshes based on the mesh-to-mesh distance metrics. The predicted muscle lengths were also compared with those manually defined on the reconstructed head and skull meshes. Moreover, the computed muscle lengths and strains were compared with those in our previous studies and the literature. Results: The skull prediction’s median deviations from the CT-based models were 2.2236 mm, 2.1371 mm, and 2.1277 mm for the skull shape, skull mesh, and muscle attachment point regions, respectively. The median deviation of the muscle lengths was 4.8940 mm. The computed muscle strains were compatible with the reported values in our previous Kinect-based method and the literature. Conclusions: The development of our novel facial muscle database opens new avenues to accurately evaluate the facial muscle states of facial palsy patients. Based on the evaluated results, specific types of facial mimic rehabilitation exercises can also be selected optimally to train the target muscles. In perspective, the database of the computed muscle lengths and strains will be integrated into our available clinical decision support system for automatically detecting malfunctioning muscles and proposing patient-specific rehabilitation serious games.
Video-audio Driven Real-Time Facial Animation.

Yilong Liu,Feng Xu,Jinxiang Chai,Xin Tong,Lijuan Wang,Qiang Huo

DOI: https://doi.org/10.1145/2816795.2818122

IF: 6.2

2015-01-01

ACM Transactions on Graphics

Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.
Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

Diego Thomas

DOI: https://doi.org/10.48550/arXiv.2004.10557

2020-04-22

Abstract:We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our proposed method is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is captured using a blendshape animation model while geometric details are captured using a Deviation image mapped over the template mesh. We contribute with an efficient algorithm to grow and refine the deforming 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion capture and 3D head modeling results on a wide range of subjects with various head poses and facial expressions.

Computer Vision and Pattern Recognition,Image and Video Processing
Non-contact Parametric Estimation and Localization of Human Head for Transcranial Magnetic Stimulation (TMS)

Yeshwin M. Srinivasa,Shaohui Foong,Dushyanth Madhavan,U-Xuan Tan,Liang Hu,Xin Fu,Yew Long Lo

DOI: https://doi.org/10.1109/ram.2013.6758591

2013-01-01

Abstract:Transcranial magnetic stimulation (TMS) has been gaining popularity in various neurological treatments and an automated platform with better precision is desired. This paper proposes a dexterous articulated robotic arm with TMS coils as the end-effectors. Most robotic arm based positioning systems utilize vision feedback from stationary cameras for dynamic positioning, but these are not suitable for clinics with limited space. In this paper, a compact approach to identify the size, position and orientation of the patient's head relative to a robotic arm using a non-vision, range data based system is proposed. An accurate distance measurement sensor is used in tandem with a robotic arm to scan the patient's head and produce a growing 3D point cloud. An efficient surface filling algorithm, taking into consideration the similarity of the human head to an ellipsoid, is presented to simultaneously extract position, orientation and geometrical information of the target head. Simulations and experiments are conducted and promising results are obtained to demonstrate its capability to identify the location and orientation of the patient's head.
Video Tracked Facial Expression Animation

L Mei,Jh Chen,Hj Bao,Qs Peng

2001-01-01

Abstract:In this paper, we present a new technique to automatically recover the face pose and facial expression from a video sequence. Based on image-based modeling technique, a personalized 3D face model is first reconstructed from two orthogonal images. The face model implanted with 18 muscular vectors is then correspondingly deformed to match each frame using a continuous optimization technique. Our method has several advantages over previous techniques. It is robust to deal with large head motions, and tracking facial features is much stable and precise over a large number of frames due to its full consideration of the 3D information of the head. The algorithm has been applied to automatically generate performance driven expression animations from video sequences.
A Method of Facial Animation Retargeting Based on Motion Capture

Dongsheng Zhou,Xiaoying Liang,Qiang Zhang,Jing Dong

DOI: https://doi.org/10.1007/978-3-319-21978-3_14

2015-01-01

Abstract:By using facial motion data from passive optical motion capture system, this paper proposed an animation reconstruction approach, which is aimed to solve the common problem: model discontinuity and reconstructing a real-like facial animation. For this problem, a facial animation reconstruction approach with model divisions is proposed. We first divide facial model into different regions and request the deformed motion data by deploying RBF mapping. Then we adjust the deformed motion data to ensure their exactness. Besides, considering the nonlinear relationship of neighbors with the current point, we also proposed a facial animation reconstruction approach without model divisions, where we apply the motion of one-ring neighbors to adjust the deformed motion data. The experiments show that the proposed methods have an acceptable precision of reconstruction.
High-fidelity facial and speech animation for VR HMDs

Kyle Olszewski,Joseph J. Lim,Shunsuke Saito,Hao Li

DOI: https://doi.org/10.1145/2980179.2980252

IF: 6.2

2016-11-11

ACM Transactions on Graphics

Abstract:Significant challenges currently prohibit expressive interaction in virtual reality (VR). Occlusions introduced by head-mounted displays (HMDs) make existing facial tracking techniques intractable, and even state-of-the-art techniques used for real-time facial tracking in unconstrained environments fail to capture subtle details of the user's facial expressions that are essential for compelling speech animation. We introduce a novel system for HMD users to control a digital avatar in real-time while producing plausible speech animation and emotional expressions. Using a monocular camera attached to an HMD, we record multiple subjects performing various facial expressions and speaking several phonetically-balanced sentences. These images are used with artist-generated animation data corresponding to these sequences to train a convolutional neural network (CNN) to regress images of a user's mouth region to the parameters that control a digital avatar. To make training this system more tractable, we use audio-based alignment techniques to map images of multiple users making the same utterance to the corresponding animation parameters. We demonstrate that this approach is also feasible for tracking the expressions around the user's eye region with an internal infrared (IR) camera, thereby enabling full facial tracking. This system requires no user-specific calibration, uses easily obtainable consumer hardware, and produces high-quality animations of speech and emotional expressions. Finally, we demonstrate the quality of our system on a variety of subjects and evaluate its performance against state-of-the-art real-time facial tracking techniques.

computer science, software engineering
High-fidelity 3D real-time facial animation using infrared structured light sensing system

Yuping Ye,Zhan Song,Juan Zhao

DOI: https://doi.org/10.1016/j.cag.2022.03.007

2022-05-01

Abstract:The acquisition of 3D facial models is crucial in the gaming and film industries. In this study, we developed a facial acquisition system based on infrared structured light sensors to obtain facial-expression models with high fidelity and accuracy. First, we employed time-multiplexing structured light to obtain an accurate and dense point cloud. The template model was then warped to the captured facial expression. A model-tracking method based on optical flow was applied to track the registered 3D model displacement. Finally, the live 3D mesh was textured using high-resolution images captured by three color cameras. Experiments were conducted on human faces to demonstrate the performance of the proposed system and methods.
Modeling short-term dynamics and variability for realistic interactive facial animation

Nicolas Stoiber,Gaspard Breton,Renaud Seguier

DOI: https://doi.org/10.1109/MCG.2010.40

Abstract:Modern modeling and rendering techniques have produced nearly photorealistic face models, but truly expressive digital faces also require natural-looking movements. Virtual characters in today's applications often display unrealistic facial expressions. Indeed, facial animation with traditional schemes such as keyframing and motion capture demands expertise. Moreover, the traditional schemes aren't adapted to interactive applications that require the real-time generation of context-dependent movements. A new animation system produces realistic expressive facial motion at interactive speed. The system relies on a set of motion models controlling facial-expression dynamics. The models are fitted on captured motion data and therefore retain the dynamic signature of human facial expressions. They also contain a nondeterministic component that ensures the variety of the long-term visual behavior. This system can efficiently animate any synthetic face. The video illustrates interactive use of a system that generates facial-animation sequences.
A Kinect-based system for cognitive rehabilitation exercises monitoring

D González-Ortega,F J Díaz-Pernas,M Martínez-Zarzuela,M Antón-Rodríguez

DOI: https://doi.org/10.1016/j.cmpb.2013.10.014

Abstract:In this paper, a 3D computer vision system for cognitive assessment and rehabilitation based on the Kinect device is presented. It is intended for individuals with body scheme dysfunctions and left-right confusion. The system processes depth information to overcome the shortcomings of a previously presented 2D vision system for the same application. It achieves left and right-hand tracking, and face and facial feature detection (eye, nose, and ears) detection. The system is easily implemented with a consumer-grade computer and an affordable Kinect device and is robust to drastic background and illumination changes. The system was tested and achieved a successful monitoring percentage of 96.28%. The automation of the human body parts motion monitoring, its analysis in relation to the psychomotor exercise indicated to the patient, and the storage of the result of the realization of a set of exercises free the rehabilitation experts of doing such demanding tasks. The vision-based system is potentially applicable to other tasks with minor changes.
Machine Learning Methods to Track Dynamic Facial Function in Facial Palsy

Akshita A. Rao,Jacqueline J. Greene,Todd P. Coleman

DOI: https://doi.org/10.1101/2024.03.16.24304357

2024-03-18

Abstract:For patients with facial paralysis, the wait for return of facial function and the resulting vision risk from poor eye closure, difficulty speaking and eating from flaccid oral sphincter muscles, as well as the psychological morbidity from the inability to smile or express emotions through facial movement can be devastating. There are limited methods to assess ongoing facial nerve regeneration: clinicians rely on subjective descriptions, imprecise scales, and static photographs to evaluate facial functional recovery and thus facial nerve regeneration remains poorly understood. We propose a more precise evaluation of dynamic facial function through video-based machine learning analysis which would facilitate a better understanding of the sometimes subtle onset of facial nerve recovery and improve guidance for facial reanimation surgery. Specifically, we here present machine learning methods employing likelihood ratio tests, optimal transport theory, and Mahalanobis distances to: 1) assess the use of defined facial landmarks for binary classification of different types of facial palsy; 2) identify regions of asymmetry and potential paralysis during specific facial cues; and 3) determining severity of abnormal facial function when compared to a reference class of normal facial function. Our work presents promising results of utilizing videos, rather than static photographs, to provide robust quantitative analyses of dynamic properties for various facial movements without requiring manual assessment. The long-term potential of this project is to enable clinicians to have more accurate and timely information to make decisions for facial reanimation surgery which will have drastic consequences on quality of life for affected patients.

Neurology
Facial performance sensing head-mounted display

Hao Li,Laura Trutoiu,Kyle Olszewski,Lingyu Wei,Tristan Trutna,Pei-Lun Hsieh,Aaron Nicholls,Chongyang Ma

DOI: https://doi.org/10.1145/2766939

IF: 6.2

2015-07-27

ACM Transactions on Graphics

Abstract:There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.

computer science, software engineering
Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution

Hyojoon Park,Sangeetha Grama Srinivasan,Matthew Cong,Doyub Kim,Byungsoo Kim,Jonathan Swartz,Ken Museth,Eftychios Sifakis

2024-09-19

Abstract:We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme.

Graphics,Artificial Intelligence
Three-dimensional Mandibular Motion Trajectory-Tracking System Based on BP Neural Network

Sukun Tian,Ning Dai,Linlin Li,Weiwei Li,Yuchun Sun,Xiaosheng Cheng

DOI: https://doi.org/10.3934/mbe.2020307

2020-01-01

Mathematical Biosciences and Engineering

Abstract:The aim of this study was to develop a prototype three-dimensional optical motion capture system based on binocular stereo vision, Back-propagation (BP) Neural Network and 3D compen-sation method for accurate and real-time recording of mandibular movement. A specialized 3D method of compensation to eliminate the involuntary vibration motions by human heart beating and respiration. A kind of binocular visual 3D measurement method based on projection line and a calibration method based on BP neural network is proposed to solve the problem of the high complexity of camera calibration process and the low accuracy of 3D measurement. The accuracy of the proposed system is systematically evaluated by means of electric platform and clinical trials, and the root-mean-square is 0.0773 mm. Finally, comparisons with state-of-the-art methods demonstrate that our system has higher reliability and accuracy. Meanwhile, the motion trajectory-tracking system is expected to be used in the diagnosis of clinical oral diseases and digital design of restoration.
Learning and Synthesizing Mpeg-4 Compatible 3-D Face Animation from Video Sequence

W Gao,YQ Chen,R Wang,SG Shan,DL Jiang

DOI: https://doi.org/10.1109/tcsvt.2003.817629

IF: 5.859

2003-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:We present a new system that applies an example-based learning method to learn facial motion patterns from a video sequence of individual facial behavior such as lip motion and facial expressions, and using that to create vivid three-dimensional (3-D) face animation according to the definition of MPEG-4 face animation parameters. The system consists of three key modules, face tracking, pattern learning, and face animation. In face tracking, to reduce the complexity of the tracking process, a novel coarse-to-fine strategy combined with a Kalman filter is proposed for localizing key facial landmarks in each image of the video. The landmarks' sequence is normalized into a visual feature matrix and then fed to the next step of process. In pattern learning, in the pretraining stage, the parameters of the camera that took the video are requested with the training video data so the system can estimate the basic mapping from a normalized two-dimensional (2-D) visual feature matrix to the representation in 3-D MPEG-4 face animation parameter space, in assistance with the computer vision method. In the practice stage, considering that in most cases camera parameters are not provided with video data, the system uses machine learning technology to complement the incomplete 3-D information for the mapping that information is needed in face orientation presentation. The example-based learning in this system integrates several methods including clustering, HMM, and ANN to make a better conversion from a 2-D to 3-D model and better estimation of incomplete 3-D information for good mapping; this will be used to drive face animation thereafter. In face animation, the system can synthesize face animation following any type of face motion in video. Experiments show that our system produces more vivid face motion animation, compared to other early systems.
Design of 3D Modeling Face Image Library in Multimedia Film and Television

Di Lei,Sae-Hoon Kim

DOI: https://doi.org/10.1155/2021/2221893

IF: 2.336

2021-10-15

Journal of Sensors

Abstract:The development of 3D modeling technology has promoted the development of the multimedia film and television industry. This article is aimed at studying the design of 3D modeling facial image library in multimedia film and television, at providing a more comprehensive facial image library for the multimedia film and television industry, at breaking the shackles of the traditional film and television industry with 3D technology, and at continuously surpassing traditional film and television media forms. This article deeply explores the background development of multimedia film and television and the characteristics of the development of new media. Starting from 3D technology, it extracts facial features of characters, transforms image data through deep autoencoders, and uses local binarization mode to perform the original facial image is subjected to texture feature extraction. In this paper, a number of experimental subjects were selected, and the subjects were photographed from the left, front, and right from multiple angles. Through the pinhole camera projection imaging process, the internal and external parameters of the camera were adjusted. In the process of 3D image construction, the image is first selected for feature detection, then the corresponding vector information and geometric conditions are matched to construct a 3D matrix, and the facial structure image is obtained by triangulation. This article compares the 3D production software on the market and selects the Maya platform suitable for building this system. The global constraint information is obtained by training some sample images. When searching the test image, find the appropriate feature point position according to the structural matching degree of the local image. When each search is completed, the global information will be used for constraint, so as to output reasonable feature information. The average residual range of the human face image constructed in this paper is 0.25-0.45, and the maximum residual error does not exceed 4.0. The experimental method in this paper has good stability and robustness. Using the COM transmission model can make experimenters not need to think too much about the underlying details. This face animation-driven simulation scheme can achieve more vivid facial expressions.

engineering, electrical & electronic,instruments & instrumentation
Early Recognition of Facial Paralysis for Rehabilitation of Stroke Patients Using Visual Perception and AI-Assisted Analysis

Yuwei Zhang,Jun Zhang,Yi Xiao,Ning Zhao,Jingyu Wu,Hao Wu,Weifeng Li,Xin Li,Ming Ma,Aiguo Song

DOI: https://doi.org/10.1109/ICARM54641.2022.9959402

2022-01-01

Abstract:Facial paralysis is one of the symptoms of neurological diseases such as stroke and Bell’s paralysis. It causes a partial loss of facial muscle control, resulting in some facial asymmetry. The traditional diagnosis needs experienced professional doctors. We propose a vision-based facial image acquisition and auxiliary diagnosis system for patients with facial paralysis. The acquisition system based on an industrial camera and array microphone collects high-quality patient images, which are preprocessed and input into the auxiliary diagnostic system for image analysis. For auxiliary diagnosis, an image classification algorithm based on fusion feature was adopted to extract Histogram of Oriented Gradients (HoG) features, facial landmark features, and learning features. The output of auxiliary diagnosis includes normal, left weakness, and right weakness. The diagnostic effect was verified on a self-made dataset. The proposed auxiliary diagnosis algorithm achieves a classification accuracy of 96.1%, recall of 91.6%, precision of 96.2%, and F2 of 92.5%. The system is suitable for auxiliary diagnosis in hospitals and early self-examination at home for the rehabilitation of stroke people. It has higher efficiency than traditional methods.
Neuromuscular Control of the Face-Head-Neck Biomechanical Complex With Learning-Based Expression Transfer From Images and Videos

Xiao S. Zeng,Surya Dwarakanath,Wuyue Lu,Masaki Nakada,Demetri Terzopoulos

DOI: https://doi.org/10.48550/arXiv.2111.06517

2021-11-12

Abstract:The transfer of facial expressions from people to 3D face models is a classic computer graphics problem. In this paper, we present a novel, learning-based approach to transferring facial expressions and head movements from images and videos to a biomechanical model of the face-head-neck complex. Leveraging the Facial Action Coding System (FACS) as an intermediate representation of the expression space, we train a deep neural network to take in FACS Action Units (AUs) and output suitable facial muscle and jaw activation signals for the musculoskeletal model. Through biomechanical simulation, the activations deform the facial soft tissues, thereby transferring the expression to the model. Our approach has advantages over previous approaches. First, the facial expressions are anatomically consistent as our biomechanical model emulates the relevant anatomy of the face, head, and neck. Second, by training the neural network using data generated from the biomechanical model itself, we eliminate the manual effort of data collection for expression transfer. The success of our approach is demonstrated through experiments involving the transfer onto our face-head-neck model of facial expressions and head poses from a range of facial images and videos.

Graphics,Computer Vision and Pattern Recognition

Partin Tables cannot accurately predict the pathological stage at radical prostatectomy.

Stereo Camera System and Its Application in Networked Virtual Environment

Mimetic Muscle Rehabilitation Analysis Using Clustering of Low Dimensional 3D Kinect Data

Novel Baseline Facial Muscle Database Using Statistical Shape Modeling and In Silico Trials toward Decision Support for Facial Rehabilitation

Video-audio Driven Real-Time Facial Animation.

Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

Non-contact Parametric Estimation and Localization of Human Head for Transcranial Magnetic Stimulation (TMS)

Video Tracked Facial Expression Animation

A Method of Facial Animation Retargeting Based on Motion Capture

High-fidelity facial and speech animation for VR HMDs

High-fidelity 3D real-time facial animation using infrared structured light sensing system

Modeling short-term dynamics and variability for realistic interactive facial animation

A Kinect-based system for cognitive rehabilitation exercises monitoring

Machine Learning Methods to Track Dynamic Facial Function in Facial Palsy

Facial performance sensing head-mounted display

Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution

Three-dimensional Mandibular Motion Trajectory-Tracking System Based on BP Neural Network

Learning and Synthesizing Mpeg-4 Compatible 3-D Face Animation from Video Sequence

Design of 3D Modeling Face Image Library in Multimedia Film and Television

Early Recognition of Facial Paralysis for Rehabilitation of Stroke Patients Using Visual Perception and AI-Assisted Analysis

Neuromuscular Control of the Face-Head-Neck Biomechanical Complex With Learning-Based Expression Transfer From Images and Videos