Abstract:The panorama stitching system is an indispensable module in surveillance or space exploration. Such a system enables the viewer to understand the surroundings instantly by aligning the surrounding images on a plane and fusing them naturally. The bottleneck of existing systems mainly lies in alignment and naturalness of the transition of adjacent images. When facing dynamic foregrounds, they may produce outputs with misaligned semantic objects, which is evident and sensitive to human perception. We solve three key issues in the existing workflow that can affect its efficiency and the quality of the obtained panoramic video and present Pedestrian360, a panoramic video system based on a structured camera array (a spatial surround-view camera system). First, to get a geometrically aligned 360○ view in the horizontal direction, we build a unified multi-camera coordinate system via a novel refinement approach that jointly optimizes camera poses. Second, to eliminate the brightness and color difference of images taken by different cameras, we design a photometric alignment approach by introducing a bias to the baseline linear adjustment model and solving it with two-step least-squares. Third, considering that the human visual system is more sensitive to high-level semantic objects, such as pedestrians and vehicles, we integrate the results of instance segmentation into the framework of dynamic programming in the seam-cutting step. To our knowledge, we are the first to introduce instance segmentation to the seam-cutting problem, which can ensure the integrity of the salient objects in a panorama. Specifically, in our surveillance oriented system, we choose the most significant target, pedestrians, as the seam avoidance target, and this accounts for the name Pedestrian360 . To validate the effectiveness and efficiency of Pedestrian360, a large-scale dataset composed of videos with pedestrians in five scenes is established. The test results on this dataset demonstrate the superiority of Pedestrian360 compared to its competitors. Experimental results show that Pedestrian360 can stitch videos at a speed of 12 to 26 fps, which depends on the number of objects in the shooting scene and their frequencies of movements. To make our reported results reproducible, the relevant code and collected data are publicly available at https://cslinzhang.github.io/Pedestrian360-Homepage/ .

Fast Construction of Dynamic and Multi-Resolution 360° Panoramas from Video Sequences

Fast Generation Of Dynamic And Multi-Resolution 360 Degrees Panorama From Video Sequences

Web-based virtual walkthrough of panoramas

An Efficient Video Mosaic Panorama Algorithm

Pedestrian-Aware Panoramic Video Stitching Based on a Structured Camera Array.

A robust image mosaicing technique capable of creating integrated panoramas

Construction 3 D Panoramic Model of Natural Scene from Real Image Sequences

Fast Video Mosaic Construction for Observation of Large Static Scenes.

Cylindrical Panoramic Mosaicing from a Pipeline Video Through MRF Based Optimization.

Multi-Viewpoint Panorama Construction with Wide-Baseline Images

A Robust Mosaic Panorama Technique for Video

Creating full view panoramic image mosaics and environment maps

Adaptive Cylindrical Panoramas Mosaic

PanoDreamer: 3D Panorama Synthesis from a Single Image

An Efficient Mosaic Panorama Technique for Medical Video

Real‐time generation of spherical panoramic video using an omnidirectional multi‐camera system

Long Scene Panorama Generation for Indoor Environment.

Constructing Big Panorama from Video Sequence Based on Deep Local Feature

Creating Cylindrical Panoramic Mosaic from a Pipeline Video

Panoramic 3d Reconstruction Using Stereo Multi-Perspective Panorama

Fast Panoramic Face Mosaicing and Recognition