Abstract:How can neural networks trained by contrastive learning extract features from the unlabeled data? Why does contrastive learning usually need much stronger data augmentations than supervised learning to ensure good representations? These questions involve both the optimization and statistical aspects of deep learning, but can hardly be answered by analyzing supervised learning, where the target functions are the highest pursuit. Indeed, in self-supervised learning, it is inevitable to relate to the optimization/generalization of neural networks to how they can encode the latent structures in the data, which we refer to as the feature learning process.
In this work, we formally study how contrastive learning learns the feature representations for neural networks by analyzing its feature learning process. We consider the case where our data are comprised of two types of features: the more semantically aligned sparse features which we want to learn from, and the other dense features we want to avoid. Theoretically, we prove that contrastive learning using $\mathbf{ReLU}$ networks provably learns the desired sparse features if proper augmentations are adopted. We present an underlying principle called $\textbf{feature decoupling}$ to explain the effects of augmentations, where we theoretically characterize how augmentations can reduce the correlations of dense features between positive samples while keeping the correlations of sparse features intact, thereby forcing the neural networks to learn from the self-supervision of sparse features. Empirically, we verified that the feature decoupling principle matches the underlying mechanism of contrastive learning in practice.
What problem does this paper attempt to address?
The core problem that this paper attempts to solve is to understand how self - supervised contrastive learning extracts features in neural networks and why contrastive learning usually requires stronger data augmentations than supervised learning to ensure good representations.
Specifically, the paper mainly explores the following issues:
1. **How to train neural networks through contrastive learning to effectively learn feature representations**: Different from supervised learning, the objective function of contrastive learning is not the ultimate pursuit, so its feature - learning process involves optimization and statistical issues. By analyzing the feature - learning process of contrastive learning, the author attempts to explain how neural networks encode the latent structure in data.
2. **Why does contrastive learning require stronger data augmentations**: In practice, contrastive learning often depends on powerful data augmentation techniques, and the specific mechanism of these augmentation techniques is still unclear. The author attempts to theoretically explain how data augmentation affects feature learning and proposes the principle of "feature decoupling" to explain the role of augmentation techniques.
### Research Contributions
The main contributions of the paper include:
- **Theoretical analysis**: Through mathematical modeling and theoretical proof, the author shows that under the sparse coding model, contrastive learning using ReLU neural networks can successfully learn the required sparse features, provided that appropriate data augmentation techniques are adopted.
- If data augmentation is not used, the neural network will learn feature representations that emphasize noise.
- Using natural augmentation techniques (such as random occlusion), the neural network can avoid learning dense noise features and focus on sparse signal features instead.
- The features learned by contrastive learning are similar to those learned by supervised learning, as long as two conditions are met: (1) The sparse signal in the data is not destroyed by augmentation; (2) The labels in supervised learning mainly depend on the sparse signal.
- **Feature decoupling principle**: The author proposes and verifies the "feature decoupling" principle, that is, data augmentation can reduce the correlation of dense features between positive samples while maintaining the correlation of sparse features, thus forcing the neural network to learn from sparse signal features.
- **Experimental evidence**: The theoretical results are verified through multiple experiments, showing the influence of different augmentation techniques on feature learning. For example, using only crop - and - zoom augmentation can only partially learn semantic features, while adding color - distortion augmentation can further remove dense color features, making edge and shape features clearer.
### Conclusion
Through theoretical analysis and experimental proof, the paper shows that the success of contrastive learning depends on data augmentation techniques. In particular, these augmentation techniques can help neural networks focus on sparse signal features through "feature decoupling" rather than being disturbed by dense noise features. This provides an important theoretical basis and empirical support for understanding and improving contrastive learning algorithms.