Joint Gaze Correction and Face Beautification for Conference Video using Dual Sparsity Prior

Deming Zhai,Xianming Liu,Xiangyang Ji,Debin Zhao,Wen Gao

DOI: https://doi.org/10.1109/tie.2018.2889616

IF: 7.7

2019-01-01

IEEE Transactions on Industrial Electronics

Abstract:Gaze mismatch is a common problem in video conferencing, which leads to that the two parties cannot converse eye-to-eye, hampering the visual communication experience. A popular approach to address this problem is to synthesize a gaze-corrected face image as viewed from the screen center via depth-image-based rendering (DIBR). Due to self-occlusion, however, there will be missing pixels in the DIBR-synthesized view image that require satisfactory filling. In this paper, we propose to jointly solve the hole-filling problem and the face beautification problem using dual sparsity prior. Specifically, prior to the start of a video conference session, we first train two dictionaries separately offline using two large datasets: one with general face images, the other with “beautiful” human faces, which means faces with high beauty scores. During the actual conference session, we solve the hole-filling and facial components beautification problems simultaneously by seeking two code vectors---one is sparse in the first dictionary and explains the available DIBR-synthesized pixels, the other is sparse in the second dictionary and matches well with the first vector in terms of feature space distance. This ensures an acceptable level of recognizability of the conference subject, while increases proximity to “beautiful” facial features to improve attractiveness. Experimental results show naturally rendered human faces with noticeably improved attractiveness.