Cross-spectral stereo matching for facial disparity estimation in the dark

Songnan Lin,Jiawei Zhang,Jing Chen,Yongtian Wang,Yicun Liu,Jimmy Ren
DOI: https://doi.org/10.1016/j.cviu.2020.103046
IF: 4.886
2020-11-01
Computer Vision and Image Understanding
Abstract:<p>Numerous applications on human faces hinge on depth information. Often, facial stereo matching provides an opportunity to estimate disparity without active projectors. However, existing algorithms are less effective at night due to unclear texture and severe noises in RGB images. In this paper, we address this problem by estimating facial disparity maps from NIR-RGB pairs. We develop a neural network composed of a multi-spectral transfer network (MSTN) and a disparity estimation network (DEN). MSTN is used to produce a pseudo-NIR image aligned with the RGB view using a spatially weighted sum on the NIR one by a kernel prediction network (KPN). As the pseudo-NIR and the NIR images share the same appearance, the facial disparity map is predicted by the proposed DEN with the same-spectral stereo pair. The whole network can be trained in an end-to-end manner and the experimental results demonstrate that it performs favorably against state-of-the-art algorithms on both synthetic and real data.</p>
computer science, artificial intelligence,engineering, electrical & electronic
What problem does this paper attempt to address?