Prune and Repaint: Content-Aware Image Retargeting for any Ratio

Feihong Shen,Chao Li,Yifeng Geng,Yongjian Deng,Hao Chen
2024-10-30
Abstract:Image retargeting is the task of adjusting the aspect ratio of images to suit different display devices or presentation environments. However, existing retargeting methods often struggle to balance the preservation of key semantics and image quality, resulting in either deformation or loss of important objects, or the introduction of local artifacts such as discontinuous pixels and inconsistent regenerated content. To address these issues, we propose a content-aware retargeting method called PruneRepaint. It incorporates semantic importance for each pixel to guide the identification of regions that need to be pruned or preserved in order to maintain key semantics. Additionally, we introduce an adaptive repainting module that selects image regions for repainting based on the distribution of pruned pixels and the proportion between foreground size and target aspect ratio, thus achieving local smoothness after pruning. By focusing on the content and structure of the foreground, our PruneRepaint approach adaptively avoids key content loss and deformation, while effectively mitigating artifacts with local repainting. We conduct experiments on the public RetargetMe benchmark and demonstrate through objective experimental results and subjective user studies that our method outperforms previous approaches in terms of preserving semantics and aesthetics, as well as better generalization across diverse aspect ratios. Codes will be available at <a class="link-external link-https" href="https://github.com/fhshen2022/PruneRepaint" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to solve two core challenges in image retargeting: 1. **Preserving Key Information**: When adjusting the aspect ratio of an image to adapt to different display devices or presentation environments, how to ensure that the key content in the image (such as important objects, scene structures, etc.) is not cropped or deformed. 2. **Avoiding Artifacts**: How to avoid local artifacts such as image distortion, broken lines, and inconsistent regenerated content caused by pixel removal or generation. Existing image retargeting methods often struggle to strike a balance between these two. For example: - **Scaling**: Although it retains all content, it causes severe deformation, reducing aesthetics and image quality, making it difficult to recognize people. - **Cropping**: It does not introduce artifacts, but often leads to the loss of key semantic information. - **Traditional pixel - shifting methods (such as seam - carving)**: Lacking semantic guidance, it is prone to cause the loss or deformation of content in important areas, as well as inconsistent pixels in the foreground. - **Deep - learning - based methods**: Although semantic information is introduced, they fail to distinguish the semantic importance within objects (for example, the face is more important than the hair), and often only focus on the foreground area, resulting in discontinuous backgrounds and a decline in aesthetics. To solve these problems, the authors propose a new content - aware image retargeting framework - **PruneRepaint**. This framework improves existing methods in the following ways: - **Content - aware Seam - carving (CSC)**: Combining hierarchical semantic information to distinguish energy, thereby pruning at the scene level and object level, maximizing the preservation of key objects and their discriminative semantic elements. - **Adaptive Repainting (AR)**: Using a diffusion model to dynamically determine whether patching or expansion is required, and performing local smoothing according to different aspect ratios to ensure the aesthetics and consistency of the image. Through these improvements, PruneRepaint can adapt to any aspect ratio while maintaining key semantics and appearance, and effectively reduce the generation of artifacts.