Non-negative Contrastive Learning

Yifei Wang,Qi Zhang,Yaoyu Guo,Yisen Wang

2024-04-23

Abstract:Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge, as these features are often opaque to human understanding. In this paper, we propose Non-negative Contrastive Learning (NCL), a renaissance of Non-negative Matrix Factorization (NMF) aimed at deriving interpretable features. The power of NCL lies in its enforcement of non-negativity constraints on features, reminiscent of NMF's capability to extract features that align closely with sample clusters. NCL not only aligns mathematically well with an NMF objective but also preserves NMF's interpretability attributes, resulting in a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCL. Empirically, we show that these advantages enable NCL to outperform CL significantly on feature disentanglement, feature selection, as well as downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning as well. Code is available at

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the issue of the lack of interpretability in deep learning representations within self-supervised learning (particularly contrastive learning). Specifically:

1. **Proposing Non-negative Contrastive Learning (NCL)**:
- Enhancing feature interpretability, sparsity, and orthogonality by introducing non-negative constraints.
- Compared to traditional Contrastive Learning (CL), NCL can generate features that are more easily understandable by humans.

2. **Theoretical Analysis and Guarantees**:
- Providing a theoretical analysis of NCL, demonstrating its advantages in feature identifiability and generalization ability for downstream tasks.
- Establishing the Bayes optimal error guarantee for NCL under ideal conditions.

3. **Experimental Validation**:
- Experimental results show that NCL outperforms CL in feature disentanglement, feature selection, and downstream classification tasks.
- NCL also demonstrates extensibility in supervised learning and introduces the Non-negative Cross-Entropy (NCE) loss.

In summary, this paper is primarily dedicated to improving the interpretability of features in self-supervised learning through non-negative contrastive learning, and it conducts both theoretical and empirical studies in multiple aspects.