Image Aesthetics Assessment With Emotion-Aware Multibranch Network
Hangwei Chen,Feng Shao,Baoyang Mu,Qiuping Jiang
DOI: https://doi.org/10.1109/tim.2024.3365174
IF: 5.6
2024-03-02
IEEE Transactions on Instrumentation and Measurement
Abstract:The aesthetic and appreciation of an image is the innate human perceptual ability. Emotion, as one of the most basic human perceptions, has been found to have a close relationship with aesthetics. However, explicitly incorporating the learned emotion cues into the image aesthetics assessment (IAA) model remains challenging. In addition, humans consider both fine-grained details and holistic context information in aesthetic assessments. Therefore, the utilization of emotional information to enhance and modulate the representation of aesthetic features in context and detail is crucial for IAA. With this motivation, we propose a new IAA method named emotion-aware multibranch network (EAMB-Net). Specifically, we first design two branches to extract aesthetic features related to detail and context. Then, an emotion branch is proposed to reveal the important emotion regions by generating the emotion-aware map (EAM). Finally, the EAM is further used to infuse emotional knowledge into the aesthetic features and enhance the feature representation, producing the final aesthetic prediction. Experimental results validate that the proposed EAMB-Net can achieve superior performance in score regression, binary classification, and score distribution tasks, obtaining the classification accuracies of 88.87% and 82.12% on the PARA and IAE datasets, respectively, using ResNet50 as the backbone. Furthermore, EAM visualization highlights the critical regions of an image, making EAMB-Net more interpretable than its competitors. Our code will be released at https://github.com/Hangwei-Chen/EAMB-Net.
engineering, electrical & electronic,instruments & instrumentation