Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning

Amin Karimi Monsefi,Mengxi Zhou,Nastaran Karimi Monsefi,Ser-Nam Lim,Wei-Lun Chao,Rajiv Ramnath

2024-09-16

Abstract:We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. Prior work in this direction masks out pre-defined frequencies in the input image and employs a reconstruction loss to pre-train the model. While achieving promising results, such an implementation has two fundamental limitations as identified in our paper. First, using pre-defined frequencies overlooks the variability of image frequency responses. Second, pre-trained with frequency-filtered images, the resulting model needs relatively more data to adapt to naturally looking images during fine-tuning. To address these drawbacks, we propose FOurier transform compression with seLf-Knowledge distillation (FOLK), integrating two dedicated ideas. First, inspired by image compression, we adaptively select the masked-out frequencies based on image frequency responses, creating more suitable SSL tasks for pre-training. Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks. Our experimental results demonstrate the effectiveness of FOLK in achieving competitive performance to many state-of-the-art SSL methods across various downstream tasks, including image classification, few-shot learning, and semantic segmentation.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address two core limitations in the Masked Frequency Modeling (MFM) method within Self-Supervised Learning (SSL):

1. **Limitations of Predefined Frequencies**: Existing MFM methods use fixed filters to mask the frequency components of input images, which ignores the variations in frequency response between different images. This makes the reconstruction task relatively simple, thereby affecting the model's training effectiveness.

2. **Lack of Sufficient Exposure to Natural Image Information**: During the pre-training phase, MFM only exposes the model to images processed with frequency masking, without a mechanism to let the model access information from the original images. This requires the pre-trained model to need more data to adapt to the natural image distribution during fine-tuning, especially performing poorly in few-shot learning scenarios.

To address these issues, the paper proposes a new framework—FOurier transform compression with se Lf-Knowledge distillation (FOLK), which enhances visual self-supervised learning through the following two major improvements:
- **Adaptive Frequency Masking**: Dynamically selects masking frequencies based on each image's unique frequency response, creating a more challenging pre-training task.
- **Knowledge Distillation Strategy**: Employs a teacher-student framework where the student model not only reconstructs missing frequencies from masked frequency images but also learns information about the original images from the feature representations generated by the teacher model, thereby improving the model's generalization ability and effectiveness in downstream tasks.