Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Hao Ai,Zidong Cao,Jinjing Zhu,Haotian Bai,Yucheng Chen,Lin Wang

2022-05-24

Abstract:Omnidirectional image (ODI) data is captured with a 360x180 field-of-view, which is much wider than the pinhole cameras and contains richer spatial information than the conventional planar images. Accordingly, omnidirectional vision has attracted booming attention due to its more advantageous performance in numerous applications, such as autonomous driving and virtual reality. In recent years, the availability of customer-level 360 cameras has made omnidirectional vision more popular, and the advance of deep learning (DL) has significantly sparked its research and applications. This paper presents a systematic and comprehensive review and analysis of the recent progress in DL methods for omnidirectional vision. Our work covers four main contents: (i) An introduction to the principle of omnidirectional imaging, the convolution methods on the ODI, and datasets to highlight the differences and difficulties compared with the 2D planar image data; (ii) A structural and hierarchical taxonomy of the DL methods for omnidirectional vision; (iii) A summarization of the latest novel learning strategies and applications; (iv) An insightful discussion of the challenges and open problems by highlighting the potential research directions to trigger more research in the community.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the challenges of omnidirectional vision in applications, particularly how to utilize deep learning (DL) techniques to process omnidirectional image (ODI) data. Specifically, the paper focuses on the following aspects:

1. **Special Properties of Omnidirectional Images**: Omnidirectional images have a 360°×180° field of view, capturing more information than traditional pinhole cameras, but also introducing severe distortion and content discontinuity issues.

2. **Application of Deep Learning Methods**: In recent years, with the proliferation of consumer-grade 360° cameras and the development of deep learning technologies, research and applications in omnidirectional vision have been significantly promoted. The paper systematically reviews and analyzes the latest advancements in deep learning within the field of omnidirectional vision.

3. **Datasets and Convolution Methods**: The paper introduces the imaging principles of omnidirectional images, convolution methods, and commonly used datasets, discussing the differences and challenges of these datasets compared to traditional 2D planar image data.

4. **Classification and Hierarchical Structure**: The paper proposes a structured classification system covering various deep learning methods in omnidirectional vision, including convolution filters, network design, novel learning strategies, and practical applications.

5. **Latest Research and Future Directions**: The paper summarizes the latest learning strategies and potential applications, delves into the challenges and unresolved issues in current research, and proposes future research directions.

Overall, the paper aims to provide a comprehensive review for researchers in the field of omnidirectional vision, helping them better understand and address the challenges in this area.