Abstract:Most Image Aesthetic Assessment (IAA) methods use a pretrained ImageNet classification model as a base to fine-tune. We hypothesize that content classification is not an optimal pretraining task for IAA, since the task discourages the extraction of features that are useful for IAA, e.g., composition, lighting, or style. On the other hand, we argue that the Contrastive Language-Image Pretraining (CLIP) model is a better base for IAA models, since it has been trained using natural language supervision. Due to the rich nature of language, CLIP needs to learn a broad range of image features that correlate with sentences describing the image content, composition, environments, and even subjective feelings about the image. While it has been shown that CLIP extracts features useful for content classification tasks, its suitability for tasks that require the extraction of style-based features like IAA has not yet been shown. We test our hypothesis by conducting a three-step study, investigating the usefulness of features extracted by CLIP compared to features obtained from the last layer of a comparable ImageNet classification model. In each step, we get more computationally expensive. First, we engineer natural language prompts that let CLIP assess an image's aesthetic without adjusting any weights in the model. To overcome the challenge that CLIP's prompting only is applicable to classification tasks, we propose a simple but effective strategy to convert multiple prompts to a continuous scalar as required when predicting an image's mean aesthetic score. Second, we train a linear regression on the AVA dataset using image features obtained by CLIP's image encoder. The resulting model outperforms a linear regression trained on features from an ImageNet classification model. It also shows competitive performance with fully fine-tuned networks based on ImageNet, while only training a single layer. Finally, by fine-tuning CLIP's image encoder on the AVA dataset, we show that CLIP only needs a fraction of training epochs to converge, while also performing better than a fine-tuned ImageNet model. Overall, our experiments suggest that CLIP is better suited as a base model for IAA methods than ImageNet pretrained networks.

Aesthetic image cropping meets VLP: Enhancing good while reducing bad

Towards Interactive Facial Image Inpainting by Text or Exemplar Image.

Image Cropping with Composition and Saliency Aware Aesthetic Score Map

CLIP knows image aesthetics

A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping

AesCLIP: Multi-Attribute Contrastive Learning for Image Aesthetics Assessment

Aesthetic-guided Outward Image Cropping

Clip-Based Composition-Aware Image Cropping

Automatic Image Cropping for Visual Aesthetic Enhancement Using Deep Neural Networks and Cascaded Regression

Repurposing existing deep networks for caption and aesthetic-guided image cropping

Exploring CLIP for Assessing the Look and Feel of Images

A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping

Deep Cropping via Attention Box Prediction and Aesthetics Assessment

Learning Image Aesthetic Assessment from Object-level Visual Components

Towards Personalized Aesthetic Image Caption

Emotion Attention-Aware Collaborative Deep Reinforcement Learning for Image Cropping

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models

Image Cropping with Spatial-Aware Feature and Rank Consistency

Image Aesthetics Assessment via Learnable Queries

An End-to-End Neural Network for Image Cropping by Learning Composition from Aesthetic Photos

Saliency Aware Image Cropping with Latent Region Pair.