Bridging the Gap Between Saliency Prediction and Image Quality Assessment

Kirillov Alexey,Andrey Moskalenko,Dmitriy Vatolin

2024-05-08

Abstract:Over the past few years, deep neural models have made considerable advances in image quality assessment (IQA). However, the underlying reasons for their success remain unclear, owing to the complex nature of deep neural networks. IQA aims to describe how the human visual system (HVS) works and to create its efficient approximations. On the other hand, Saliency Prediction task aims to emulate HVS via determining areas of visual interest. Thus, we believe that saliency plays a crucial role in human perception. In this work, we conduct an empirical study that reveals the relation between IQA and Saliency Prediction tasks, demonstrating that the former incorporates knowledge of the latter. Moreover, we introduce a novel SACID dataset of saliency-aware compressed images and conduct a large-scale comparison of classic and neural-based IQA methods. All supplementary code and data will be available at the time of publication.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The paper primarily explores the relationship between Image Quality Assessment (IQA) and Saliency Prediction (SP), and attempts to address the following key questions:

1. **Investigate whether IQA models inherently understand saliency in the Human Visual System (HVS)**: Researchers verify whether these models already incorporate an understanding of saliency in their prediction process by extracting saliency maps from IQA models.

2. **Develop a parameter-free dual-task training strategy**: This strategy can simultaneously optimize both IQA and saliency prediction tasks without degrading the overall performance of the model.

3. **Create a new dataset to evaluate the quality of non-uniformly compressed images**: Researchers constructed a dataset named SACID, which includes non-uniformly compressed images, to evaluate the performance of different IQA methods under this specific compression mode.

4. **Evaluate the effectiveness of existing IQA metrics**: Through a subjective study involving more than 1400 evaluators, the effectiveness of existing IQA metrics for non-uniformly compressed content using various saliency-aware coding strategies is assessed.

In summary, the main goal of this paper is to reveal the connection between IQA and saliency prediction, and to empirically demonstrate the importance of saliency in human visual perception and how it can be applied to enhance the performance of IQA models. Additionally, the paper proposes new methods and techniques to improve current IQA evaluation standards and shows the potential limitations of existing IQA metrics in non-uniform compression scenarios.