Steadiface: Real-Time Face-Centric Stabilization on Mobile Phones

Fuhao Shi,Sung-Fang Tsai,Youyou Wang,Chia-Kai Liang

DOI: https://doi.org/10.48550/arXiv.1905.01382

2019-05-04

Abstract:We present Steadiface, a new real-time face-centric video stabilization method that simultaneously removes hand shake and keeps subject's head stable. We use a CNN to estimate the face landmarks and use them to optimize a stabilized head center. We then formulate an optimization problem to find a virtual camera pose that locates the face to the stabilized head center while retains smooth rotation and translation transitions across frames. We test the proposed method on fieldtest videos and show it stabilizes both the head motion and background. It is robust to large head pose, occlusion, facial appearance variations, and different kinds of camera motions. We show our method advances the state of art in selfie video stabilization by comparing against alternative methods. The whole process runs very efficiently on a modern mobile phone (8.1 ms/frame).

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the instability of human faces in selfie videos. Specifically, existing video stabilization methods are not effective in handling selfie videos. Especially in the case of hand - shaking and head movement, the stability of the human face cannot be effectively guaranteed. To solve this problem, the author proposes a new method named Steadiface, which aims to eliminate hand - shaking and keep the head stable simultaneously.

### Main problems:
1. **Instability of human faces caused by hand - shaking and head movement**: When recording selfie videos, unintentional hand - shaking and head movement will cause the human face to appear unstable in the video.
2. **Limitations of existing methods**: Most existing video stabilization methods cannot handle selfie videos well. Especially when the difference between face movement and background movement is large, the human face may still be unstable.

### Solutions:
Steadiface solves the above problems in the following ways:
- **Real - time facial center stabilization**: Use a convolutional neural network (CNN) to estimate facial key points and optimize a stable head center based on these key points.
- **Virtual camera pose optimization**: Find a virtual camera pose through an optimization problem, position the human face at the stable head center, and maintain a smooth transition between frames in terms of rotation and translation.
- **Robustness**: This method is robust to a wide range of head poses, occlusions, changes in facial appearance, and different types of camera movements.

### Key contributions:
1. **The first real - time end - to - end stabilization system**: It can stabilize head and camera movements simultaneously.
2. **Novel algorithm**: Combine gyroscope stability and facial stability into a single objective function for joint optimization.
3. **Effective weight adjustment scheme**: Dynamically adjust the weights in the optimization process to deal with head pose changes and noisy landmark positions.
4. **Extensive comparative experiments**: Conducted extensive comparisons with existing state - of - the - art methods, demonstrating its superior performance.

Through these innovations, Steadiface can run efficiently on mobile devices (only 8.1 milliseconds per frame) and significantly improve the stability of human faces in selfie videos.