MirrorCalib: Utilizing Human Pose Information for Mirror-based Virtual Camera Calibration

Longyun Liao,Rong Zheng,Andrew Mitchell

2024-05-18

Abstract:In this paper, we present the novel task of estimating the extrinsic parameters of a virtual camera relative to a real camera in exercise videos with a mirror. This task poses a significant challenge in scenarios where the views from the real and mirrored cameras have no overlap or share salient features. To address this issue, prior knowledge of a human body and 2D joint locations are utilized to estimate the camera extrinsic parameters when a person is in front of a mirror. We devise a modified eight-point algorithm to obtain an initial estimation from 2D joint locations. The 2D joint locations are then refined subject to human body constraints. Finally, a RANSAC algorithm is employed to remove outliers by comparing their epipolar distances to a predetermined threshold. MirrorCalib achieves a rotation error of 1.82° and a translation error of 69.51 mm on a collected real-world dataset, which outperforms the state-of-art method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper proposes a new method called MirrorCalib to solve the problem of estimating the extrinsic parameters of a virtual camera (mirror) relative to a real camera in motion tutorial videos. In these scenarios, it is difficult to find shared features from the views of the real camera and the mirror camera due to the presence of a mirror. To address this challenge, MirrorCalib utilizes human joint information and obtains initial estimation of 2D joint positions using a modified 8-point algorithm. It then optimizes the 2D joint positions by considering anatomical constraints of the human body and uses the Random Sample Consensus (RANSAC) algorithm to exclude outliers. This approach achieves a rotation error of 1.82° and a translation error of 69.51 millimeters on a real-world dataset, outperforming existing techniques.

What sets MirrorCalib apart is that it only requires a view of a fixed mirror, without the need for a multiple mirror system, making it more suitable for everyday environments such as gyms and dance studios. Additionally, unlike traditional methods that rely on visible feature correspondences, MirrorCalib uses 2D joint positions of the human body, which are usually not visible, as corresponding features. Although there are other works that use 2D human joints to calibrate the extrinsic parameters of real cameras, MirrorCalib jointly optimizes joint position estimation and virtual camera calibration, thereby improving accuracy.