Causal Perception Inspired Representation Learning for Trustworthy Image Quality Assessment

Lei Wang,Desen Yuan

2024-04-30

Abstract:Despite great success in modeling visual perception, deep neural network based image quality assessment (IQA) still remains unreliable in real-world applications due to its vulnerability to adversarial perturbations and the inexplicit black-box structure. In this paper, we propose to build a trustworthy IQA model via Causal Perception inspired Representation Learning (CPRL), and a score reflection attack method for IQA model. More specifically, we assume that each image is composed of Causal Perception Representation (CPR) and non-causal perception representation (N-CPR). CPR serves as the causation of the subjective quality label, which is invariant to the imperceptible adversarial perturbations. Inversely, N-CPR presents spurious associations with the subjective quality label, which may significantly change with the adversarial perturbations. To extract the CPR from each input image, we develop a soft ranking based channel-wise activation function to mediate the causally sufficient (beneficial for high prediction accuracy) and necessary (beneficial for high robustness) deep features, and based on intervention employ minimax game to optimize. Experiments on four benchmark databases show that the proposed CPRL method outperforms many state-of-the-art adversarial defense methods and provides explicit model interpretation.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

### The Problems This Paper Attempts to Solve

This paper aims to address two main issues in the field of Image Quality Assessment (IQA):

1. **Vulnerability to Adversarial Attacks**: Existing IQA models based on deep neural networks are prone to erroneous predictions when faced with adversarial perturbations. Even very small, imperceptible perturbations can significantly alter the model's scoring results, making the model unreliable.

2. **Lack of Interpretability in Black-Box Structures**: Current IQA models are often black-box structures, making it difficult to explain their internal mechanisms, which reduces the model's reliability in practical applications.

To address these issues, the authors propose a method based on Causal Perception inspired Representation Learning (CPRL). Specifically, CPRL enhances the robustness and interpretability of the model through the following means:
- Dividing the image into Causal Perception Representations (CPR) and Non-Causal Perception Representations (N-CPR).
- CPR, as the cause of subjective quality labels, is invariant to imperceptible adversarial perturbations.
- N-CPR, on the other hand, has a spurious correlation with subjective quality labels and may undergo significant changes under adversarial perturbations.

By extracting CPR from each input image, the authors developed a soft-sorting-based channel activation function to coordinate causally sufficient features (which contribute to high prediction accuracy) and necessary features (which contribute to high robustness), and optimized it through intervention using a minimax game.

Experimental results show that the proposed CPRL method outperforms many existing adversarial defense methods on four benchmark databases and provides clear model explanations.