Abstract:Mid-level vision capabilities - such as generic object localization and 3D geometric understanding - are not only fundamental to human vision but are also crucial for many real-world applications of computer vision. These abilities emerge with minimal supervision during the early stages of human visual development. Despite their significance, current self-supervised learning (SSL) approaches are primarily designed and evaluated for high-level recognition tasks, leaving their mid-level vision capabilities largely unexamined. In this study, we introduce a suite of benchmark protocols to systematically assess mid-level vision capabilities and present a comprehensive, controlled evaluation of 22 prominent SSL models across 8 mid-level vision tasks. Our experiments reveal a weak correlation between mid-level and high-level task performance. We also identify several SSL methods with highly imbalanced performance across mid-level and high-level capabilities, as well as some that excel in both. Additionally, we investigate key factors contributing to mid-level vision performance, such as pretraining objectives and network architectures. Our study provides a holistic and timely view of what SSL models have learned, complementing existing research that primarily focuses on high-level vision tasks. We hope our findings guide future SSL research to benchmark models not only on high-level vision tasks but on mid-level as well.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to explore and evaluate the performance of self - supervised learning (SSL) models in mid - level vision tasks. Specifically, the researchers focus on the following issues: 1. **Is the mid - level vision ability of SSL models related to the high - level vision ability?** - The researchers want to know whether an SSL model that performs excellently in high - level vision tasks (such as image classification, object detection, etc.) also has equally excellent performance in mid - level vision tasks (such as general object segmentation, depth estimation, surface normal estimation, etc.). 2. **What factors make SSL models perform well in mid - level vision tasks?** - The researchers hope to identify the key factors that affect the mid - level vision ability of SSL models, such as pre - training objectives, network architectures, etc. 3. **How do existing SSL models perform on mid - level vision tasks?** - The researchers systematically evaluated the performance of 22 popular SSL models on 8 mid - level vision tasks to reveal the actual capabilities of these models in such tasks. ### Main contributions of the paper - **Introduced a set of benchmark test protocols**: for systematically evaluating the mid - level vision ability of SSL models. - **Comprehensively evaluated multiple SSL models**: covering different categories and generations of SSL methods, providing in - depth understanding of the mid - level vision ability of these models. - **Discovered the phenomenon of performance imbalance**: there are significant differences in the performance of some SSL models between mid - level and high - level vision tasks. - **Revealed the factors affecting mid - level vision ability**: including pre - training objectives, network architectures, and model capacity, etc. ### Conclusion Through this research, the authors hope to guide future research to focus not only on high - level vision tasks but also more on mid - level vision tasks, thereby promoting the development of self - supervised learning models in a wider range of application scenarios. In addition, the research results also provide valuable references for designing more effective SSL models. ### Formula display To ensure the correctness and readability of the formulas, the following are some formula examples involved in the paper (in Markdown format): - **Binary Cross - Entropy Loss**: \[ \text{BCE}(y, \hat{y}) = -\frac{1}{N} \sum_{i = 1}^{N} \left[ y_i \log(\hat{y}_i)+(1 - y_i) \log(1 - \hat{y}_i) \right] \] - **Root Mean Squared Error (RMSE)**: \[ \text{RMSE}=\sqrt{\frac{1}{N} \sum_{i = 1}^{N} (y_i - \hat{y}_i)^2} \] - **Cosine Similarity**: \[ \text{Cosine Similarity}(x, y)=\frac{x \cdot y}{\|x\| \|y\|} \] - **Angular Error**: \[ \text{Angular Error}=\arccos\left(\frac{x \cdot y}{\|x\| \|y\|}\right) \] These formulas help readers better understand the technical details and evaluation indicators involved in the paper.

Probing the Mid-level Vision Capabilities of Self-Supervised Learning

Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection.

Learning Where to Learn in Cross-View Self-Supervised Learning

Self-Supervised Learning in Remote Sensing: A review

Progress and Thinking on Self-Supervised Learning Methods in Computer Vision: A Review

A Survey of the Self Supervised Learning Mechanisms for Vision Transformers

Semi-Supervised and Unsupervised Deep Visual Learning: A Survey

Index Your Position: A Novel Self-Supervised Learning Method for Remote Sensing Images Semantic Segmentation

A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification

Self-supervised visual learning in the low-data regime: a comparative evaluation

A generic self-supervised learning (SSL) framework for representation learning from spectra-spatial feature of unlabeled remote sensing imagery

CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

On the Discriminability of Self-Supervised Representation Learning

Understanding and Improving the Role of Projection Head in Self-Supervised Learning

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

A Survey of Self-Supervised Learning from Multiple Perspectives: Algorithms, Theory, Applications and Future Trends

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

Unleash Model Potential: Bootstrapped Meta Self-supervised Learning

Investigating Self-Supervised Methods for Label-Efficient Learning

Learning What and Where to Learn: A New Perspective on Self-supervised Learning