Content-Preserving Image Stitching with Regular Boundary Constraints

Yun Zhang,Yu-Kun Lai,Fang-Lue Zhang
DOI: https://doi.org/10.48550/arXiv.1810.11220
2019-02-11
Abstract:This paper proposes an approach to content-preserving stitching of images with regular boundary constraints, which aims to stitch multiple images to generate a panoramic image with regular boundary. Existing methods treat image stitching and rectangling as two separate steps, which may result in suboptimal results as the stitching process is not aware of the further warping needs for rectangling. We address these limitations by formulating image stitching with regular boundaries in a unified optimization. Starting from the initial stitching results produced by traditional warping-based optimization, we obtain the irregular boundary from the warped meshes by polygon Boolean operations which robustly handle arbitrary mesh compositions, and by analyzing the irregular boundary construct a piecewise rectangular boundary. Based on this, we further incorporate straight line preserving and regular boundary constraints into the image stitching framework, and conduct iterative optimization to obtain an optimal piecewise rectangular boundary, thus can make the panoramic boundary as close as possible to a rectangle, while reducing unwanted distortions. We further extend our method to panoramic videos and selfie photography, by integrating the temporal coherence and portrait preservation into the optimization. Experiments show that our method efficiently produces visually pleasing panoramas with regular boundaries and unnoticeable distortions.
Graphics
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to generate panoramas with regular boundaries (especially rectangular boundaries) during the image stitching process. Existing methods usually treat image stitching and rectangularization as two independent steps, which may lead to sub - optimal results because the stitching process does not take into account the subsequent rectangularization requirements. Specifically, when dealing with images taken by hand - held cameras, existing methods may cause severe distortion or content loss if the traditional stitching method is directly used due to the large parallax and irregular boundaries of these images. To overcome these problems, this paper proposes a new method that simultaneously performs image stitching and regular - boundary optimization through a unified optimization framework, thereby reducing unnecessary distortion and preserving the content of the input images as much as possible. The main contributions of this method include: 1. **Unified Optimization Framework**: A global optimization method is proposed, which reduces the distortion problem caused by step - by - step processing in traditional methods by simultaneously performing image stitching and boundary regularization. 2. **Piecewise Rectangular Boundary**: The concept of piecewise rectangular boundary is introduced to avoid excessive distortion while maintaining boundary regularity. This method can better handle situations where large areas are missing in the scene. 3. **Automatic Algorithm**: A fully automatic algorithm is developed to generate optimized piecewise rectangular boundaries, balancing distortion and boundary simplicity. Through these innovations, this method can generate visually satisfactory panoramas with regular boundaries and minimal distortion, thereby enhancing the viewing experience of panoramas. In addition, this method can also be extended to selfie - extension and video - stitching applications.