Abstract:This paper shows that Masking the Deep hierarchical features is an efficient self-supervised method, denoted as MaskDeep. MaskDeep treats each patch in the representation space as an independent instance. We mask part of patches in the representation space and then utilize sparse visible patches to reconstruct high semantic image representation. The intuition of MaskDeep lies in the fact that models can reason from sparse visible patches semantic to the global semantic of the image. We further propose three designs in our framework: 1) a Hierarchical Deep-Masking module to concern the hierarchical property of patch representations, 2) a multi-group strategy to improve the efficiency without any extra computing consumption of the encoder and 3) a multi-target strategy to provide more description of the global semantic. Our MaskDeep brings decent improvements. Trained on ResNet50 with 200 epochs, MaskDeep achieves state-of-the-art results of 71.2% Top1 accuracy linear classification on ImageNet. On COCO object detection tasks, MaskDeep outperforms the self-supervised method SoCo, which specifically designed for object detection. When trained with 100 epochs, MaskDeep achieves 69.6% Top1 accuracy, which surpasses current methods trained with 200 epochs, such as HCSC, by 0.4% .

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to effectively use local information to construct global image representations in self - supervised learning. Specifically, the authors propose a new self - supervised learning method - MaskDeep. By masking deep - level features in the feature space and using sparsely visible local features to reconstruct high - semantic image representations. #### Main problems: 1. **Improve the effectiveness of self - supervised learning**: Existing self - supervised learning methods such as contrastive learning and masked image modeling (MIM) perform well on some tasks, but there is still room for improvement. In particular, how to improve model performance while reducing computational overhead is an important challenge. 2. **Integrate local and global information**: Many existing methods focus either on global information or on local information, and lack a method that can effectively integrate both simultaneously. The goal of this paper is to enable the model to infer global semantics from sparse local features through reasonable framework design. #### Solutions: - **MaskDeep**: This method masks patches in the representation space and uses sparsely visible patches to reconstruct high - level image representations. The core idea is to infer global image content through reasoning about the relationships between local features. - **Hierarchical Deep - Masking Module**: To consider the hierarchy of feature representations, a hierarchical deep - masking module based on the FPN module is introduced to generate a hierarchical representation space. - **Multi - group Strategy**: By sampling sparse local features multiple times without increasing the computational consumption of the encoder, the training efficiency is improved. - **Multi - target Strategy**: By having the joint patch representation predict multiple global targets, more supervision signals are provided to maintain the stability of training. ### Experimental results: - On the ImageNet linear classification task, after 200 epochs of training, MaskDeep achieved a Top - 1 accuracy of 71.2%, surpassing current self - supervised learning methods. - On the COCO object detection task, MaskDeep also performed well, outperforming SoCo, a self - supervised method specifically designed for object detection. In summary, this paper solves the problem of how to efficiently use local information to construct global image representations by introducing MaskDeep and achieves significant performance improvements on multiple downstream tasks.

Mask Hierarchical Features For Self-Supervised Learning

Masked Autoencoders for Point Cloud Self-supervised Learning.

Evolved Hierarchical Masking for Self-Supervised Learning

When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

Masked Feature Modeling for Generative Self-Supervised Representation Learning of High-Resolution Remote Sensing Images

Evolved Part Masking for Self-Supervised Learning

Hierarchical Superpixel Relation Graph Combined with Convolutional Sparse Coding for Self-Supervised Hyperspectral Image Denoising

Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition

Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

Masked Face Recognition with Generative-to-Discriminative Representations

Self-Supervised Medical Image Segmentation Using Deep Reinforced Adaptive Masking

Mask-R-FCN: A Deep Fusion Network for Semantic Segmentation.

Masked Autoencoders in 3D Point Cloud Representation Learning

Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution

Instructional Mask Autoencoder: A Scalable Learner for Hyperspectral Image Classification

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Hierarchical Dynamic Masks for Visual Explanation of Neural Networks

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

Rethinking mask heads for partially supervised instance segmentation

Masked Image Modeling with Local Multi-Scale Reconstruction.

Convolutional Feature Masking for Joint Object and Stuff Segmentation