Abstract:Many leading self-supervised methods for unsupervised representation learning, in particular those for embedding image features, are built on variants of the instance discrimination task, whose optimization is known to be prone to instabilities that can lead to feature collapse. Different techniques have been devised to circumvent this issue, including the use of negative pairs with different contrastive losses, the use of external memory banks, and breaking of symmetry by using separate encoding networks with possibly different structures. Our method, termed BAM, rather than directly matching features of different views (augmentations) of input images, is based on matching their self-attention vectors, which are the distributions of similarities to the entire set of augmented images of a batch. We obtain rich representations and avoid feature collapse by minimizing a loss that matches these distributions to their globally balanced and entropy regularized version, which is obtained through a simple self-optimal-transport computation. We ablate and verify our method through a wide set of experiments that show competitive performance with leading methods on both semi-supervised and transfer-learning benchmarks. Our implementation and pre-trained models are available at <a class="link-external link-http" href="http://github.com/DanielShalam/BAM" rel="external noopener nofollow">this http URL</a> .

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the **feature collapse problem in unsupervised representation learning**. Specifically, many leading self - supervised methods rely on the instance discrimination task when performing image feature embedding, and the optimization process of these tasks is prone to instability, resulting in feature collapse. Feature collapse means that the feature representations learned by the model become single or meaningless and cannot effectively distinguish different data samples. To solve this problem, the authors propose a method named BAM (Balanced Self - Attention Matching). Instead of directly matching the feature vectors of different views (i.e., different augmented versions), BAM avoids feature collapse by matching the self - attention vectors of these views. The self - attention vector reflects the similarity distribution between each augmented image and other augmented images in the entire batch. BAM achieves this by minimizing a loss function that matches these distributions with their globally balanced and entropy - regularized versions. This method can not only obtain rich feature representations, but also effectively avoid feature collapse. ### Main Contributions 1. **Introducing Self - Attention Matching**: Avoid feature collapse by matching the self - attention vectors of different views instead of directly matching the feature vectors. 2. **Global Balancing and Entropy Regularization**: Ensure the diversity and stability of feature representations by performing global balancing and entropy regularization on the self - attention matrix. 3. **Experimental Verification**: Through extensive experimental verification, it is proved that the performance of BAM on semi - supervised learning and transfer learning benchmarks is better than existing methods. ### Formula Summary - Calculation of the self - attention matrix \(A\): \[ A=\text{SM}_\tau(S) \] where \(S\) is the matrix of all pairwise cosine similarities, and \(\text{SM}_\tau\) is the softmax operation with the temperature parameter \(\tau\). - Calculation of the balance matrix \(B\): \[ B = \arg\min_{B'\in B_{n\cdot k}}\langle - S, B'\rangle-\tau h(B') \] where \(B_{n\cdot k}\) is the set of \(n\cdot k\times n\cdot k\) doubly stochastic matrices, and \(h(M)=-\sum_{i,j}m_{ij}\log(m_{ij})\) is the Shannon entropy of the matrix \(M\). - BAM loss function: \[ L_{\text{BAM}}=\sum_{j\neq j'}\sum_{i = 1}^n\text{CE}(b_j^i,a_{j'}^i)=-\sum_{j\neq j'}\sum_{i = 1}^n\langle b_j^i,\log a_{j'}^i\rangle \] Through these methods, BAM has achieved remarkable results in unsupervised representation learning, especially in avoiding feature collapse.

Unsupervised Representation Learning by Balanced Self Attention Matching

ActiveMatch: End-to-end Semi-supervised Active Representation Learning

Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

Self-Supervised Node Representation Learning Via Node-to-Neighbourhood Alignment.

Align Yourself: Self-supervised Pre-training for Fine-grained Recognition via Saliency Alignment.

Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations

Self-Supervised Representation Learning with Meta Comprehensive Regularization

Unsupervised Representation Learning to Aid Semi-Supervised Meta Learning

Self-Supervised Representation Learning for Adversarial Attack Detection

The Common Stability Mechanism behind most Self-Supervised Learning Approaches

Adaptive Similarity Bootstrapping for Self-Distillation based Representation Learning

Semantic Positive Pairs for Enhancing Visual Representation Learning of Instance Discrimination methods

Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins

Multi-Task Self-Supervised Time-Series Representation Learning

Contrastive Self-supervised Representation Learning Using Synthetic Data

Unsupervised Document Embedding via Contrastive Augmentation

Self-supervised Learning is More Robust to Dataset Imbalance

Self-reinforcing Unsupervised Matching

Mitigating background bias in self-supervised video representation learning

Align Representations with Base: A New Approach to Self-Supervised Learning

From Pretext to Purpose: Batch-Adaptive Self-Supervised Learning