RecDiffusion: Rectangling for Image Stitching with Diffusion Models

Tianhao Zhou,Haipeng Li,Ziyi Wang,Ao Luo,Chen-Lin Zhang,Jiajun Li,Bing Zeng,Shuaicheng Liu

2024-03-28

Abstract:Image stitching from different captures often results in non-rectangular boundaries, which is often considered unappealing. To solve non-rectangular boundaries, current solutions involve cropping, which discards image content, inpainting, which can introduce unrelated content, or warping, which can distort non-linear features and introduce artifacts. To overcome these issues, we introduce a novel diffusion-based learning framework, \textbf{RecDiffusion}, for image stitching rectangling. This framework combines Motion Diffusion Models (MDM) to generate motion fields, effectively transitioning from the stitched image's irregular borders to a geometrically corrected intermediary. Followed by Content Diffusion Models (CDM) for image detail refinement. Notably, our sampling process utilizes a weighted map to identify regions needing correction during each iteration of CDM. Our RecDiffusion ensures geometric accuracy and overall visual appeal, surpassing all previous methods in both quantitative and qualitative measures when evaluated on public benchmarks. Code is released at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the issue of non-rectangular boundaries that arise after image stitching. Specifically:

1. **Problems with Existing Methods**:
- **Cropping**: Simply cropping out the non-rectangular edges results in a reduced field of view and loss of some image content.
- **Inpainting**: Using inpainting techniques (such as Stable Diffusion) may introduce additional content that is unrelated to the original image.
- **Warping**: Using warping techniques may lead to nonlinear feature distortion and artifacts.

2. **Proposed Solution**:
- The authors propose a new framework based on Diffusion Models—RecDiffusion, to handle the non-rectangular boundary issue after image stitching.
- This framework combines Motion Diffusion Models (MDM) and Content Diffusion Models (CDM) to generate high-quality rectangular boundary images.
- MDM is used to generate motion fields, transforming the stitched image from irregular boundaries to geometrically corrected intermediate results; CDM is used to refine image details, ensuring geometric accuracy and overall visual quality.

### Main Contributions

1. **Proposed the First Diffusion Model-Based Framework**: RecDiffusion, for handling non-rectangular boundary issues after image stitching.
2. **Proposed MDM**: To generate rectangularized motion fields, followed by CDM to further refine image details.
3. **Experimental Results**: The method outperforms previous methods in both quantitative and qualitative evaluations on public benchmarks.

### Conclusion

The paper aims to address the issue of non-rectangular boundaries after image stitching using diffusion models, proposing a novel framework RecDiffusion, and validates its superior performance through extensive experiments.