Directorial Editing: A Hybrid Deep-Learning Approach to Content-Aware Image Retargeting and Resizing

Elliot Dickman,Paul Diefenbach

DOI: https://doi.org/10.3390/electronics13224459

IF: 2.9

2024-11-16

Electronics

Abstract:Image retargeting is a common computer graphics task which involves manipulating the size or aspect ratio of an image. This task often presents a challenge to the artist or user, because manipulating the size of an image necessitates some degree of data loss as pixels need to be removed to accommodate a different image size. We present an image retargeting framework which implements a confidence map generated by a segmentation model for content-aware resizing, allowing users to specify which subjects in an image to preserve using natural language prompts much like the role of an art director conversing with their artist. Using computer vision models to detect object positions also provides additional control over the composition of the retargeted image at various points in the image-processing pipeline. This object-based approach to energy map augmentation is incredibly flexible, because only minor adjustments to the processing of the energy maps can provide a significant degree of control over where seams—paths of pixels through the image—are removed, and how seam removal is prioritized in different sections of the image. It also provides additional control with techniques for object and background separation and recomposition. This research explores how several different types of deep-learning models can be integrated into this pipeline in order to easily make these decisions, and provide different retargeting results on the same image based on user input and compositional considerations. Because this is a framework based on existing machine-learning models, this approach will benefit from advancements in the rapidly developing fields of computer vision and large language models and can be extended for further natural language directorial controls over images.

engineering, electrical & electronic,computer science, information systems,physics, applied