Abstract:The success of self-supervised learning (SSL) has been the focus of multiple recent theoretical and empirical studies, including the role of data augmentation (in feature decoupling) as well as complete and dimensional representation collapse. While complete collapse is well-studied and addressed, dimensional collapse has only gain attention and addressed in recent years mostly using variants of redundancy reduction (aka whitening) techniques. In this paper, we further explore a complementary approach to whitening via feature decoupling for improved representation learning while avoiding representation collapse. In particular, we perform feature decoupling by early promotion of useful features via careful feature coloring. The coloring technique is developed based on a Bayesian prior of the augmented data, which is inherently encoded for feature decoupling. We show that our proposed framework is complementary to the state-of-the-art techniques, while outperforming both contrastive and recent non-contrastive methods. We also study the different effects of coloring approach to formulate it as a general complementary technique along with other baselines.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are the feature decoupling and representation collapse problems in self - supervised learning (SSL). Specifically: 1. **Feature decoupling**: - The core idea of self - supervised learning is to train the model through data augmentation to complete a pretext task. However, not all data augmentation methods can effectively decouple useful sparse features and less useful dense features. Therefore, how to effectively promote feature decoupling through augmentation techniques is a key issue. 2. **Representation collapse**: - Representation collapse means that during the SSL training process, the model may learn some trivial representations, causing all features to become the same or highly correlated. This includes complete collapse, that is, all features become constants; and dimensional collapse, that is, some dimensions are highly correlated, reducing the effective dimension of the representation space. - Although existing techniques such as carefully designed training protocols and asymmetric architecture designs can deal with complete collapse, the dimensional collapse problem still remains challenging. Although the existing whitening techniques can partially solve this problem, they may also limit the model's capacity, especially in the low - level feature learning stage. To solve these problems, the paper proposes a method based on coloring transform to further enhance feature decoupling and avoid representation collapse. The main contributions of this method are as follows: - **Propose a coloring - transform - based technique**, which enhances feature decoupling by introducing meaningful correlations (coloring) while eliminating unnecessary redundancies. - **Develop a direct coloring technique**, which utilizes Bayesian prior and does not require the traditional whitening step, thus achieving faster coloring transformation. - **Conduct detailed empirical studies**, showing that the coloring technique is effective not only for whitening - based SSL frameworks but also can improve other non - whitening baseline methods. Through these methods, the paper aims to improve the performance of self - supervised learning models, accelerate learning convergence, and avoid the occurrence of complete collapse.

Direct Coloring for Self-Supervised Enhanced Feature Decoupling

On Feature Decorrelation in Self-Supervised Learning

Understanding Whitening Loss in Self-supervised Learning

Self-driven Dual-path Learning for Reference-based Line Art Colorization under Limited Data

An Investigation into Whitening Loss for Self-supervised Learning

DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders

Colorization as a Proxy Task for Visual Understanding

Mind Your Augmentation: the Key to Decoupling Dense Self-Supervised Learning

Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation

On the Discriminability of Self-Supervised Representation Learning

DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning

Disentanglement of Color and Shape Representations for Continual Learning

Colour augmentation for improved semi-supervised semantic segmentation

Modulate Your Spectrum in Self-Supervised Learning

Deep Semi-supervised Learning with Double-Contrast of Features and Semantics

A Tale of Color Variants: Representation and Self-Supervised Learning in Fashion E-Commerce

Rethinking Self-Supervised Learning Within the Framework of Partial Information Decomposition

Decoupled Self-Supervised Label Augmentation for Fully-Supervised Image Classification.

Color-$S^{4}L$: Self-supervised Semi-supervised Learning with Image Colorization

De-coupling and De-positioning Dense Self-supervised Learning

Understanding self-supervised Learning Dynamics without Contrastive Pairs