Real-time 3D neural facial animation from binocular video

Chen Cao,Vasu Agrawal,Fernando De La Torre,Lele Chen,Jason Saragih,Tomas Simon,Yaser Sheikh

DOI: https://doi.org/10.1145/3476576.3476645

IF: 6.2

2021-08-01

ACM Transactions on Graphics

Abstract:ACM Transactions on Graphics, <a href="https://dl.acm.org/toc/tog/2021/40/4">Volume 40, Issue 4</a>, Page 1-17, August 2021. <br/>

computer science, software engineering

What problem does this paper attempt to address?

Real-time Facial Animation with Image-Based Dynamic Avatars.

Chen Cao,Hongzhi Wu,Yanlin Weng,Tianjia Shao,Kun Zhou

DOI: https://doi.org/10.1145/2897824.2925873

IF: 6.2

2016-01-01

ACM Transactions on Graphics

Abstract:We present a novel image-based representation for dynamic 3D avatars, which allows effective handling of various hairstyles and headwear, and can generate expressive facial animations with finescale details in real-time. We develop algorithms for creating an image-based avatar from a set of sparsely captured images of a user, using an off-the-shelf web camera at home. An optimization method is proposed to construct a topologically consistent morphable model that approximates the dynamic hair geometry in the captured images. We also design a real-time algorithm for synthesizing novel views of an image-based avatar, so that the avatar follows the facial motions of an arbitrary actor. Compelling results from our pipeline are demonstrated on a variety of cases.
Real-Time Facial Segmentation and Performance Capture from RGB Input

Shunsuke Saito,Tianye Li,Hao Li

DOI: https://doi.org/10.48550/arXiv.1604.02647

2016-04-10

Abstract:We introduce the concept of unconstrained real-time 3D facial performance capture through explicit semantic segmentation in the RGB input. To ensure robustness, cutting edge supervised learning approaches rely on large training datasets of face images captured in the wild. While impressive tracking quality has been demonstrated for faces that are largely visible, any occlusion due to hair, accessories, or hand-to-face gestures would result in significant visual artifacts and loss of tracking accuracy. The modeling of occlusions has been mostly avoided due to its immense space of appearance variability. To address this curse of high dimensionality, we perform tracking in unconstrained images assuming non-face regions can be fully masked out. Along with recent breakthroughs in deep learning, we demonstrate that pixel-level facial segmentation is possible in real-time by repurposing convolutional neural networks designed originally for general semantic segmentation. We develop an efficient architecture based on a two-stream deconvolution network with complementary characteristics, and introduce carefully designed training samples and data augmentation strategies for improved segmentation accuracy and robustness. We adopt a state-of-the-art regression-based facial tracking framework with segmented face images as training, and demonstrate accurate and uninterrupted facial performance capture in the presence of extreme occlusion and even side views. Furthermore, the resulting segmentation can be directly used to composite partial 3D face models on the input images and enable seamless facial manipulation tasks, such as virtual make-up or face replacement.

Computer Vision and Pattern Recognition
Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation

Chen Cao,Qiming Hou,Kun Zhou

DOI: https://doi.org/10.1145/2601097.2601204

IF: 6.2

2014-01-01

ACM Transactions on Graphics

Abstract:We present a fully automatic approach to real-time facial tracking and animation with a single video camera. Our approach does not need any calibration for each individual user. It learns a generic regressor from public image datasets, which can be applied to any user and arbitrary video cameras to infer accurate 2D facial landmarks as well as the 3D facial shape from 2D video frames. The inferred 2D landmarks are then used to adapt the camera matrix and the user identity to better match the facial expressions of the current user. The regression and adaptation are performed in an alternating manner. With more and more facial expressions observed in the video, the whole process converges quickly with accurate facial tracking and animation. In experiments, our approach demonstrates a level of robustness and accuracy on par with state-of-the-art techniques that require a time-consuming calibration step for each individual user, while running at 28 fps on average. We consider our approach to be an attractive solution for wide deployment in consumer-level applications.
3D Shape Regression for Real-Time Facial Animation

Chen Cao,Yanlin Weng,Stephen Lin,Kun Zhou

DOI: https://doi.org/10.1145/2461912.2462012

IF: 6.2

2013-01-01

ACM Transactions on Graphics

Abstract:We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated.
Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Bangbang Yang,Yinda Zhang,Yijin Li,Zhaopeng Cui,Sean Fanello,Hujun Bao,Guofeng Zhang

DOI: https://doi.org/10.1145/3528223.3530163

IF: 6.2

2022-01-01

ACM Transactions on Graphics

Abstract:We, as human beings, can understand and picture a familiar scene from arbitrary viewpoints given a single image, whereas this is still a grand challenge for computers. We hereby present a novel solution to mimic such human perception capability based on a new paradigm of a modal 3D scene understanding with neural rendering for a closed scene. Specifically, we first learn the prior knowledge of the objects in a closed scene via an offline stage, which facilitates an online stage to understand the room with unseen furniture arrangement. During the online stage, given a panoramic image of the scene in different layouts, we utilize a holistic neural-rendering-based optimization framework to efficiently estimate the correct 3D scene layout and deliver realistic free-viewpoint rendering. In order to handle the domain gap between the offline and online stage, our method exploits compositional neural rendering techniques for data augmentation in the offline training. The experiments on both synthetic and real datasets demonstrate that our two-stage design achieves robust 3D scene understanding and outperforms competing methods by a large margin, and we also show that our realistic free-viewpoint rendering enables various applications, including scene touring and editing. Code and data are available on the project webpage: https://zju3dv.github.io/nr_in_a_room/.
Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks

Samuli Laine,Tero Karras,Timo Aila,Antti Herva,Shunsuke Saito,Ronald Yu,Hao Li,Jaakko Lehtinen

DOI: https://doi.org/10.48550/arXiv.1609.06536

2017-06-02

Abstract:We present a real-time deep learning framework for video-based facial performance capture -- the dense 3D tracking of an actor's face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5-10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system can drastically reduce the amount of labor involved in the development of modern narrative-driven video games or films involving realistic digital doubles of actors and potentially hours of animated dialogue per character. We compare our results with several state-of-the-art monocular real-time facial capture techniques and demonstrate compelling animation inference in challenging areas such as eyes and lips.

Computer Vision and Pattern Recognition,Graphics
Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

Diego Thomas

DOI: https://doi.org/10.48550/arXiv.2004.10557

2020-04-22

Abstract:We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our proposed method is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is captured using a blendshape animation model while geometric details are captured using a Deviation image mapped over the template mesh. We contribute with an efficient algorithm to grow and refine the deforming 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion capture and 3D head modeling results on a wide range of subjects with various head poses and facial expressions.

Computer Vision and Pattern Recognition,Image and Video Processing
High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

Yunpeng Bai,Yanbo Fan,Xuan Wang,Yong Zhang,Jingxiang Sun,Chun Yuan,Ying Shan

DOI: https://doi.org/10.48550/arXiv.2211.15064

2023-03-04

Abstract:High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depend on a conditional deformation field for dynamic modeling, we propose to learn a personalized generative prior, which is formulated as a local and low dimensional subspace in the latent space of 3D-GAN. We propose an efficient method to construct the personalized generative prior based on a small set of facial images of a given individual. After learning, it allows for photo-realistic rendering with novel views and the face reenactment can be realized by performing navigation in the latent space. Our proposed method is applicable for different driven signals, including RGB images, 3DMM coefficients, and audios. Compared with existing works, we obtain superior novel view synthesis results and faithfully face reenactment performance.

Computer Vision and Pattern Recognition
Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

Philipp Ladwig,Rene Ebertowski,Alexander Pech,Ralf Dörner,Christian Geiger

2024-09-19

Abstract:While head-mounted displays (HMDs) for Virtual Reality (VR) have become widely available in the consumer market, they pose a considerable obstacle for a realistic face-to-face conversation in VR since HMDs hide a significant portion of the participants faces. Even with image streams from cameras directly attached to an HMD, stitching together a convincing image of an entire face remains a challenging task because of extreme capture angles and strong lens distortions due to a wide field of view. Compared to the long line of research in VR, reconstruction of faces hidden beneath an HMD is a very recent topic of research. While the current state-of-the-art solutions demonstrate photo-realistic 3D reconstruction results, they require high-cost laboratory equipment and large computational costs. We present an approach that focuses on low-cost hardware and can be used on a commodity gaming computer with a single GPU. We leverage the benefits of an end-to-end pipeline by means of Generative Adversarial Networks (GAN). Our GAN produces a frontal-facing 2.5D point cloud based on a training dataset captured with an RGBD camera. In our approach, the training process is offline, while the reconstruction runs in real-time. Our results show adequate reconstruction quality within the 'learned' expressions. Expressions not learned by the network produce artifacts and can trigger the Uncanny Valley effect.

Computer Vision and Pattern Recognition
Video-audio Driven Real-Time Facial Animation.

Yilong Liu,Feng Xu,Jinxiang Chai,Xin Tong,Lijuan Wang,Qiang Huo

DOI: https://doi.org/10.1145/2816795.2818122

IF: 6.2

2015-01-01

ACM Transactions on Graphics

Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.
Real-Time 3D Eye Performance Reconstruction for RGBD Cameras

F. Xu,Q. Wen,J. Yong

DOI: https://doi.org/10.1109/TVCG.2016.2641442

IF: 5.2

2017-12-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:This paper proposes a real-time method for 3D eye performance reconstruction using a single RGBD sensor. Combined with facial surface tracking, our method generates more pleasing facial performance with vivid eye motions. In our method, a novel scheme is proposed to estimate eyeball motions by minimizing the differences between a rendered eyeball and the recorded image. Our method considers and handles different appearances of human irises, lighting variations and highlights on images via the proposed eyeball model and the <inline-formula><tex-math notation="LaTeX"> $L0$</tex-math><alternatives><inline-graphic xlink:href="wen-ieq1-2641442.gif"/></alternatives></inline-formula>-based optimization. Robustness and real-time optimization are achieved through the novel 3D Taylor expansion-based linearization. Furthermore, we propose an online bidirectional regression method to handle occlusions and other tracking failures on either of the two eyes from the information of the opposite eye. Experiments demonstrate that our technique achieves robust and accurate eye performance reconstruction for different iris appearances, with various head/face/eye motions, and under different lighting conditions.

Medicine,Engineering,Computer Science
Neural monocular 3D human motion capture with physical awareness

Soshi Shimada,Vladislav Golyanik,Weipeng Xu,Patrick Pérez,Christian Theobalt

DOI: https://doi.org/10.1145/3476576.3476640

IF: 6.2

2021-08-01

ACM Transactions on Graphics

Abstract:ACM Transactions on Graphics, <a href="https://dl.acm.org/toc/tog/2021/40/4">Volume 40, Issue 4</a>, Page 1-15, August 2021. <br/>

computer science, software engineering
Real-Time 3D Video Synthesis from Binocular Stereo Camera

Xiubing Xu,Xudong Xie,Qionghai Dai

DOI: https://doi.org/10.1109/3DTV.2008.4547826

2008-01-01

Abstract:In this paper, an efficient method to synthesize 3D video in real-time from binocular stereo camera is presented. Our stereo camera consists of two pre-calibrated cameras with the optical axis perpendicular to the camera displacement. While much work has been done on stereo processing recently, most of them are focused on high-quality depth map generation, which are usually time-consuming and unsuitable for on-line processing. In our method, we apply a dynamic programming based stereo matching algorithm to compute the dense depth map on the fly. A fast depth image based rendering algorithm is subsequently developed to render several virtual views. Compared to other similar work, our method can synthesize 3D video of good performance in real-time on commodity PCs. Besides, benefiting from the well organized system architecture, our scheme is flexible for various 3D displays. Experimental results show that our method enables effective synthesis of 3D video in real-time.
Video-Driven Neural Physically-Based Facial Asset for Production

Longwen Zhang,Chuxiao Zeng,Qixuan Zhang,Hongyang Lin,Ruixiang Cao,Wei Yang,Lan Xu,Jingyi Yu

DOI: https://doi.org/10.1145/3550454.3555445

IF: 6.2

2022-11-30

ACM Transactions on Graphics

Abstract:Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.

computer science, software engineering
Real-time 3D-aware Portrait Video Relighting

Ziqi Cai,Kaiwen Jiang,Shu-Yu Chen,Yu-Kun Lai,Hongbo Fu,Boxin Shi,Lin Gao

2024-10-24

Abstract:Synthesizing realistic videos of talking faces under custom lighting conditions and viewing angles benefits various downstream applications like video conferencing. However, most existing relighting methods are either time-consuming or unable to adjust the viewpoints. In this paper, we present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF). Given an input portrait video, our method can synthesize talking faces under both novel views and novel lighting conditions with a photo-realistic and disentangled 3D representation. Specifically, we infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. We also leverage a temporal consistency network to ensure smooth transitions and reduce flickering artifacts. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed. We demonstrate the effectiveness and interactivity of our method on various portrait videos with diverse lighting and viewing conditions.

Computer Vision and Pattern Recognition,Graphics
Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

Yury Kartynnik,Artsiom Ablavatski,Ivan Grishchenko,Matthias Grundmann

DOI: https://doi.org/10.48550/arXiv.1907.06724

2019-07-15

Computer Vision and Pattern Recognition

Abstract:We present an end-to-end neural network-based model for inferring an approximate 3D mesh representation of a human face from single camera input for AR applications. The relatively dense mesh model of 468 vertices is well-suited for face-based AR effects. The proposed model demonstrates super-realtime inference speed on mobile GPUs (100-1000+ FPS, depending on the device and model variant) and a high prediction quality that is comparable to the variance in manual annotations of the same image.
Real Time System for Facial Analysis

Janne Tommola,Pedram Ghazi,Bishwo Adhikari,Heikki Huttunen

DOI: https://doi.org/10.48550/arXiv.1809.05474

2018-09-14

Computer Vision and Pattern Recognition

Abstract:In this paper we describe the anatomy of a real-time facial analysis system. The system recognizes the age, gender and facial expression from users in appearing in front of the camera. All components are based on convolutional neural networks, whose accuracy we study on commonly used training and evaluation sets. A key contribution of the work is the description of the interplay between processing threads for frame grabbing, face detection and the three types of recognition. The python code for executing the system uses common libraries--keras/tensorflow, opencv and dlib--and is available for download.
BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis

Hao-Bin Duan,Miao Wang,Jin-Chuan Shi,Xu-Chuan Chen,Yan-Pei Cao

DOI: https://doi.org/10.1145/3618399

IF: 6.2

2023-12-05

ACM Transactions on Graphics

Abstract:Synthesizing photorealistic 4D human head avatars from videos is essential for VR/AR, telepresence, and video game applications. Although existing Neural Radiance Fields (NeRF)-based methods achieve high-fidelity results, the computational expense limits their use in real-time applications. To overcome this limitation, we introduce BakedAvatar , a novel representation for real-time neural head avatar synthesis, deployable in a standard polygon rasterization pipeline. Our approach extracts deformable multi-layer meshes from learned isosurfaces of the head and computes expression-, pose-, and view-dependent appearances that can be baked into static textures for efficient rasterization. We thus propose a three-stage pipeline for neural head avatar synthesis, which includes learning continuous deformation, manifold, and radiance fields, extracting layered meshes and textures, and fine-tuning texture details with differential rasterization. Experimental results demonstrate that our representation generates synthesis results of comparable quality to other state-of-the-art methods while significantly reducing the inference time required. We further showcase various head avatar synthesis results from monocular videos, including view synthesis, face reenactment, expression editing, and pose editing, all at interactive frame rates on commodity devices. Source codes and demos are available on our project page.

computer science, software engineering
Real-Time Multi-Task Facial Analytics With Event Cameras

Cian Ryan,Amr Elrasad,Waseem Shariff,Joe Lemley,Paul Kielty,Patrick Hurney,Peter Corcoran

DOI: https://doi.org/10.1109/access.2023.3297500

IF: 3.9

2023-08-02

IEEE Access

Abstract:Event cameras, unlike traditional frame-based cameras, excel in detecting and reporting changes in light intensity on a per-pixel basis. This unique technology offers numerous advantages, including high temporal resolution, low latency, wide dynamic range, and reduced power consumption. These characteristics make event cameras particularly well-suited for sensing applications such as monitoring drivers or human behavior. This paper presents a feasibility study on the using a multitask neural network with event cameras for real-time facial analytics. Our proposed network simultaneously estimates head pose, eye gaze, and facial occlusions. Notably, the network is trained on synthetic event camera data, and its performance is demonstrated and validated using real event data in real-time driving scenarios. To compensate for global head motion, we introduce a novel event integration method capable of handling both short and long-term temporal dependencies. The performance of our facial analytics method is quantitatively evaluated in both controlled lab environments and unconstrained driving scenarios. The results demonstrate that useful accuracy and computational speed is achieved by the proposed method to determining head pose and relative eye-gaze direction. This shows that neuromorphic facial analytics can be realized in real-time and are well-suited for edge/embedded computing deployments. While the improvement ratio in comparison to existing literature may not be as favorable due to the unique event-based vision approach employed, it is crucial to note that our research focuses specifically on event-based vision, which offers distinct advantages over traditional RGB vision. Overall, this study contributes to the emerging field of event-based vision systems and highlights the potential of multitask neural networks combined with event cameras for real-time sensing of human subjects. These techniques can be applied in practical applications such as driver monitoring s- stems, interactive human-computer systems and for human behavior analysis.

computer science, information systems,telecommunications,engineering, electrical & electronic
AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels

Yuelang Xu,Lizhen Wang,Xiaochen Zhao,Hongwen Zhang,Yebin Liu

DOI: https://doi.org/10.1145/3588432.3591567

2023-05-03

Abstract:With NeRF widely used for facial reenactment, recent methods can recover photo-realistic 3D head avatar from just a monocular video. Unfortunately, the training process of the NeRF-based methods is quite time-consuming, as MLP used in the NeRF-based methods is inefficient and requires too many iterations to converge. To overcome this problem, we propose AvatarMAV, a fast 3D head avatar reconstruction method using Motion-Aware Neural Voxels. AvatarMAV is the first to model both the canonical appearance and the decoupled expression motion by neural voxels for head avatar. In particular, the motion-aware neural voxels is generated from the weighted concatenation of multiple 4D tensors. The 4D tensors semantically correspond one-to-one with 3DMM expression basis and share the same weights as 3DMM expression coefficients. Benefiting from our novel representation, the proposed AvatarMAV can recover photo-realistic head avatars in just 5 minutes (implemented with pure PyTorch), which is significantly faster than the state-of-the-art facial reenactment methods. Project page: <a class="link-external link-https" href="https://www.liuyebin.com/avatarmav" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

Real-time 3D neural facial animation from binocular video

Real-time Facial Animation with Image-Based Dynamic Avatars.

Real-Time Facial Segmentation and Performance Capture from RGB Input

Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation

3D Shape Regression for Real-Time Facial Animation

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks

Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

Video-audio Driven Real-Time Facial Animation.

Real-Time 3D Eye Performance Reconstruction for RGBD Cameras

Neural monocular 3D human motion capture with physical awareness

Real-Time 3D Video Synthesis from Binocular Stereo Camera

Video-Driven Neural Physically-Based Facial Asset for Production

Real-time 3D-aware Portrait Video Relighting

Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

Real Time System for Facial Analysis

BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis

Real-Time Multi-Task Facial Analytics With Event Cameras

AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels