Abstract:Out-of-distribution (OOD) detection is crucial for ensuring the reliable deployment of deep models in real-world scenarios. Recently, from the perspective of over-parameterization, a series of methods leveraging weight sparsification techniques have shown promising performance. These methods typically focus on selecting important parameters for in-distribution (ID) data to reduce the negative impact of redundant parameters on OOD detection. However, we empirically find that these selected parameters may behave overconfidently toward OOD data and hurt OOD detection. To address this issue, we propose a simple yet effective post-hoc method called Instance-aware Test Pruning (ITP), which performs OOD detection by considering both coarse-grained and fine-grained levels of parameter pruning. Specifically, ITP first estimates the class-specific parameter contribution distribution by exploring the ID data. By using the contribution distribution, ITP conducts coarse-grained pruning to eliminate redundant parameters. More importantly, ITP further adopts a fine-grained test pruning process based on the right-tailed Z-score test, which can adaptively remove instance-level overconfident parameters. Finally, ITP derives OOD scores from the pruned model to achieve more reliable predictions. Extensive experiments on widely adopted benchmarks verify the effectiveness of ITP, demonstrating its competitive performance.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the reliability problem of deep - learning models when processing **Out - of - Distribution (OOD) data**. Specifically, the author focuses on how to improve the performance of OOD detection through parameter pruning techniques.
#### Background and problem description
1. **The importance of OOD detection**:
- In real - world applications, such as autonomous driving and medical diagnosis, deep neural networks (DNNs) may make over - confident but wrong predictions on OOD data. This unreliable behavior may lead to catastrophic consequences.
- Therefore, accurately detecting and rejecting OOD inputs is crucial for ensuring the reliability of AI systems.
2. **Limitations of existing methods**:
- Many existing OOD detection methods rely on model outputs or gradients to design OOD scoring functions, but these methods ignore the problem that DNNs are usually over - parameterized.
- Over - parameterization makes the model more vulnerable to redundant parameter noise, resulting in the fragility of OOD detection.
- Although some post - processing methods based on sparsification (such as DICE, LINe) reduce the interference of redundant parameters by selecting important parameters, these methods may make the model show too much confidence in OOD data, thus affecting the accuracy of OOD detection.
#### Proposed method: Instance - aware Test Pruning (ITP)
To solve the above problems, the author proposes **Instance - aware Test Pruning (ITP)**, which is a simple and effective post - processing method aiming to improve the performance of OOD detection through coarse - grained and fine - grained parameter pruning.
1. **Coarse - Grained Redundancy Pruning (CRP)**:
- CRP reduces noise interference by estimating the class - specific parameter contribution distribution and removing the redundant parameters that contribute the least to the ID data prediction.
- Specifically, CRP ranks the parameters according to their average contribution to the ID data prediction and prunes the p% of parameters with the lowest contribution.
2. **Fine - Grained Test Pruning (FTP)**:
- FTP customizes the pruning pattern for each sample at test time and removes the over - confident parameters with abnormally high contribution to ID prediction through the right - tailed Z - score test.
- This method can adaptively prevent the abnormal increase of ID confidence caused by over - confident parameters, thereby reducing the risk of the model making confident but wrong predictions.
3. **Overall effect**:
- By combining CRP and FTP, ITP can more reliably distinguish between ID and OOD data and improve the performance of OOD detection.
#### Experimental results
The experimental results show that ITP significantly outperforms existing OOD detection methods on multiple benchmark datasets, especially on large - scale datasets such as CIFAR and ImageNet. For example, on CIFAR - 100, ITP reduces FPR95 by 33.42%, and also achieves the best FPR95 performance on ImageNet.
In conclusion, this paper solves the performance degradation problems in existing OOD detection methods caused by redundant parameters and over - confident parameters by proposing ITP, thereby improving the reliability and accuracy of the model when processing OOD data.