Abstract:Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics. Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

A Learning-Based Approach For Perceptual Models Of Preference

A Probabilistic Approach for Extracting Design Preferences from Design Team Discussion

Human-In-The-Loop Learning of Qualitative Preference Models

PERCEPTUAL USER INTERFACE IN VIRTUAL SHOPPING ENVIRONMENT

A Hybrid Model of Requirement Acquisition Based on Consumer's Preferenceon for 3D Printing Cloud Service Platform

Learning Personalized Exploration in Evolutionary Design Using Aesthetic Descriptors

MVReward: Better Aligning and Evaluating Multi-View Diffusion Models with Human Preferences

Optimal Design for Human Feedback

PRIMA++: A Probabilistic Framework for User Choice Modelling with Small Data.

Color Aesthetics: Fuzzy based User-driven Method for Harmony and Preference Prediction

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

A Survey on Human Preference Learning for Large Language Models

Spatially Small-scale Approach-avoidance Behaviors Allow Learning-free Machine Inference of Object Preferences in Human Minds

Data-Centric Human Preference Optimization with Rationales

Critiquing-based Modeling of Subjective Preferences

Learning to see people like people

Form + Function: Optimizing Aesthetic Product Design via Adaptive, Geometrized Preference Elicitation

Engineering Deep Representations for Modeling Aesthetic Perception

Data-driven preference learning methods for value-driven multiple criteria sorting with interacting criteria

3D computational modeling and perceptual analysis of kinetic depth effects

Active Preference-based Learning for Multi-dimensional Personalization