3DFlowRenderer: One-shot Face Re-enactment via Dense 3D Facial Flow Estimation

Siddharth Nijhawan,Takuya Yashima,Tamaki Kojima

2024-04-23

Abstract:Performing facial expression transfer under one-shot setting has been increasing in popularity among research community with a focus on precise control of expressions. Existing techniques showcase compelling results in perceiving expressions, but they lack robustness with extreme head poses. They also struggle to accurately reconstruct background details, thus hindering the realism. In this paper, we propose a novel warping technology which integrates the advantages of both 2D and 3D methods to achieve robust face re-enactment. We generate dense 3D facial flow fields in feature space to warp an input image based on target expressions without depth information. This enables explicit 3D geometric control for re-enacting misaligned source and target faces. We regularize the motion estimation capability of the 3D flow prediction network through proposed "Cyclic warp loss" by converting warped 3D features back into 2D RGB space. To ensure the generation of finer facial region with natural-background, our framework only renders the facial foreground region first and learns to inpaint the blank area which needs to be filled due to source face translation, thus reconstructing the detailed background without any unwanted pixel motion. Extensive evaluation reveals that our method outperforms state-of-the-art techniques in rendering artifact-free facial images.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of one-shot face re-enactment, particularly the precise control and background detail reconstruction under extreme head poses. While existing techniques perform well in perceiving expressions, they lack robustness under extreme head poses and struggle to accurately reconstruct background details, affecting realism. This paper proposes a new deformation technique that combines the advantages of 2D and 3D methods by generating dense 3D facial flow fields to achieve robust face re-enactment.

Specifically, the method deforms the input image in the feature space using 3D facial flow fields, allowing operations based on target expressions without requiring depth information. This approach enables explicit 3D geometric control of misaligned faces between the source and target images. Additionally, to ensure the generation of more refined facial regions and natural backgrounds, the framework first renders only the facial foreground region and learns to fill in the blank areas caused by the movement of the source face, thereby reconstructing detailed backgrounds without any unwanted pixel movements. Experimental results demonstrate that this method outperforms existing techniques in rendering artifact-free facial images.