Kang Liao,Lang Nie,Shujuan Huang,Chunyu Lin,Jing Zhang,Yao Zhao,Moncef Gabbouj,Dacheng Tao
Abstract:Camera calibration involves estimating camera parameters to infer geometric features from captured sequences, which is crucial for computer vision and robotics. However, conventional calibration is laborious and requires dedicated collection. Recent efforts show that learning-based solutions have the potential to be used in place of the repeatability works of manual calibrations. Among these solutions, various learning strategies, networks, geometric priors, and datasets have been investigated. In this paper, we provide a comprehensive survey of learning-based camera calibration techniques, by analyzing their strengths and limitations. Our main calibration categories include the standard pinhole camera model, distortion camera model, cross-view model, and cross-sensor model, following the research trend and extended applications. As there is no benchmark in this community, we collect a holistic calibration dataset that can serve as a public platform to evaluate the generalization of existing methods. It comprises both synthetic and real-world data, with images and videos captured by different cameras in diverse scenes. Toward the end of this paper, we discuss the challenges and provide further research directions. To our knowledge, this is the first survey for the learning-based camera calibration (spanned 8 years). The summarized methods, datasets, and benchmarks are available and will be regularly updated at <a class="link-external link-https" href="https://github.com/KangLiao929/Awesome-Deep-Camera-Calibration" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The main focus of this paper is to improve the process of camera calibration and its related applications using deep learning techniques, and it provides a comprehensive review of research in this field. Specifically, the paper addresses the following points:
1. **Importance and Challenges of Camera Calibration**: Camera calibration is a fundamental task in computer vision and robotics, used to estimate camera parameters to infer geometric features extracted from captured sequences. Traditional methods often require manual operation, making the process cumbersome and difficult to automate.
2. **Application of Deep Learning in Camera Calibration**: In recent years, learning-based methods, especially deep learning methods, have been proven to effectively replace traditional manual calibration work. These methods can automatically complete the calibration process without the need for specific targets or complex pre-settings.
3. **Different Types of Camera Calibration**: The paper covers calibration techniques for various types of cameras, including standard pinhole camera models, distortion camera models, cross-view models, and cross-sensor models, and analyzes the advantages and limitations of these techniques.
4. **Summary and Evaluation of Existing Work**: The paper organizes over 100 related research papers, classifies them according to different calibration goals and application scenarios, and discusses the differences in network architecture, loss functions, datasets, evaluation metrics, and other aspects of various methods.
5. **Construction of Public Datasets**: To promote further research and development in this field, the authors have also collected a comprehensive calibration dataset, including synthetic images and real-world image and video data, which can be used to evaluate the generalization ability of existing methods.
6. **Future Research Directions**: Finally, the paper discusses the challenges faced by current learning-based camera calibration and proposes possible research directions, providing guidance for future scholars.
In summary, this paper provides a comprehensive review of learning-based camera calibration techniques, offering readers the latest advancements in the field and pointing out directions for future research.