Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customization

Jooyeol Yun,Jaegul Choo

2024-10-16

Abstract:The task of personalized image aesthetic assessment seeks to tailor aesthetic score prediction models to match individual preferences with just a few user-provided inputs. However, the scalability and generalization capabilities of current approaches are considerably restricted by their reliance on an expensive curated database. To overcome this long-standing scalability challenge, we present a unique approach that leverages readily available databases for general image aesthetic assessment and image quality assessment. Specifically, we view each database as a distinct image score regression task that exhibits varying degrees of personalization potential. By determining optimal combinations of task vectors, known to represent specific traits of each database, we successfully create personalized models for individuals. This approach of integrating multiple models allows us to harness a substantial amount of data. Our extensive experiments demonstrate the effectiveness of our approach in generalizing to previously unseen domains-a challenge previous approaches have struggled to achieve-making it highly applicable to real-world scenarios. Our novel approach significantly advances the field by offering scalable solutions for personalized aesthetic assessment and establishing high standards for future research. <a class="link-external link-https" href="https://yeolj00.github.io/personal-projects/personalized-aesthetics/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problems of scalability and generalization ability in the field of Personalized Image Aesthetic Assessment (PIAA). Specifically, the existing PIAA methods are limited in scalability and generalization ability to unseen domains, mainly because these methods rely on expensive and manually - curated datasets. To overcome this long - standing scalability challenge, the authors propose a new method that utilizes off - the - shelf General Image Aesthetic Assessment (GIAA) and Image Quality Assessment (IQA) databases to create personalized models.

### Main contributions:
1. **Data - rich PIAA method**: This method addresses the long - standing scalability challenge and no longer relies on expensive manually - curated databases but utilizes existing GIAA and IQA databases.
2. **Excellent performance in cross - database evaluation**: This method demonstrates strong generalization ability in cross - database evaluation, surpassing existing methods and setting a high standard for future research.
3. **Parameter - efficient personalized method**: By learning the optimal combination of task vectors, this method can achieve accurate personalization in a highly parameter - efficient manner, taking advantage of the comprehensive information embedded in the task vectors.

### Method overview:
- **Acquisition of task vectors**: Train multiple models from multiple databases, with each model corresponding to a specific task. The task vector is defined as the difference between the fine - tuned weights and the pre - trained weights.
- **Learning the optimal combination**: Introduce learnable coefficients to combine task vectors so that the model can better match the user's aesthetic preferences. Use the ranking loss function to optimize these coefficients.
- **Adaptive coefficient initialization**: Based on zero - shot personalization performance, use the Spearman Rank - Order Correlation Coefficient (SROCC) to initialize the coefficients to improve training efficiency.

### Experimental results:
- **Cross - database evaluation**: The experimental results on the AADB and REAL - CUR databases show that this method significantly outperforms existing methods under the 10 - shot and 100 - shot protocols, and even exceeds the model trained directly on the target database.
- **In - database evaluation**: Further experiments on the Flickr - AES database verify the effectiveness of this method, and its performance is better than existing meta - learning techniques and methods that combine user attributes.

### Conclusion:
The method proposed in this paper not only makes significant progress in scalability and generalization ability but also provides a new solution for personalized image aesthetic assessment, laying the foundation for future research.