Abstract:Self-supervised pre-training medical foundation models on large-scale datasets demonstrate exceptional performance. However, recent research questions this traditional notion, exploring whether an increase in pre-training data always leads to enhanced model performance. To address this issue, data-effective learning approaches have been introduced to select valuable samples for foundation model pre-training. Notably, current methods in this area lack a clear standard for sample selection, and the underlying theoretical foundation remains unknown. As the first attempt to address this limitation, we leverage V-information in self-supervised pre-training of foundation models. Our theoretical derivation confirms that by optimizing V-information, sample selection can be framed as an optimization problem where choosing diverse and challenging samples enhances model performance even under limited training data. Under this guidance, we develop an optimal data-effective learning method (OptiDEL) to optimize V-information in real-world medical domains. The OptiDEL method generates more diverse and harder samples to achieve or even exceed the performance of models trained on the full dataset while using substantially less data. We compare the OptiDEL method with state-of-the-art approaches finding that OptiDEL consistently outperforms existing approaches across eight different datasets, with foundation models trained on only 5% of the pre-training data surpassing the performance of those trained on the full dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: when self - supervised pre - training medical basic models, how to select valuable samples by optimizing V - information, so as to improve the model performance with limited training data. Specifically, the paper explores whether increasing the amount of pre - training data can always improve the model performance, and proposes a new method - OptiDEL (Optimal Data - Effective Learning) to optimize V - information and generate more diverse and challenging samples, thereby achieving or exceeding the model performance trained with the complete data set. ### Problem Background In recent years, self - supervised pre - training medical basic models on large - scale data sets have demonstrated excellent performance. However, recent research has begun to question this traditional concept and explore whether increasing pre - training data can always bring performance improvement. For this reason, researchers have introduced data - effective learning methods, aiming to select valuable samples from large - scale data sets for pre - training. But current methods lack clear sample selection criteria and theoretical basis. ### Core Problems of the Paper 1. **Lack of clear sample selection criteria**: Current data - effective learning methods lack a unified standard when selecting pre - training samples. 2. **Insufficient theoretical basis**: The theoretical basis of these methods has not been fully established, especially in the field of self - supervised learning. 3. **How to improve model performance with limited data**: How to optimize V - information to achieve or exceed the model performance trained with the complete data set with limited pre - training data. ### Solutions The paper first introduces V - information into self - supervised pre - training tasks, regarding it as a conditional entropy optimization problem. Through theoretical derivation, the author proves that selecting more diverse and more challenging samples can optimize V - information, thereby improving the model performance. Based on this, they propose the OptiDEL method, which uses the Segment Anything Model (SAM) to extract key information and generate more diverse and challenging pre - training samples. ### Main Contributions 1. **Optimizing V - information**: For the first time, the task of self - supervised pre - training medical basic models is formalized as the problem of optimizing V - information. Through theoretical derivation, it is proved that generating more diverse data and selecting the most challenging samples are helpful for optimizing V - information. 2. **Designing the OptiDEL method**: Based on V - information, the OptiDEL method is designed. By extracting key data information, more diverse and more challenging samples are generated, thereby enhancing the performance of the basic model. 3. **Experimental verification**: OptiDEL is compared with the existing state - of - the - art MedDEL method on eight real - world medical data sets. The experimental results show the importance of optimizing V - information for improving the performance of data - effective learning. Through these contributions, the paper provides a new and effective data - effective learning method for self - supervised pre - training medical basic models, which can achieve or exceed the model performance trained with the complete data set with limited data.

Optimizing V-information for Self-Supervised Pre-training Data-Effective Medical Foundation Models

A Medical Data-Effective Learning Benchmark for Highly Efficient Pre-training of Foundation Models

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

Medical image foundation models in assisting diagnosis of brain tumors: a pilot study

MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification

Improved Fine-Tuning by Better Leveraging Pre-Training Data

Foundation AI Model for Medical Image Segmentation

Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

On the Challenges and Perspectives of Foundation Models for Medical Image Analysis

Towards General Purpose Medical AI: Continual Learning Medical Foundation Model

Specialized Foundation Models Struggle to Beat Supervised Baselines

A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models

Improving Representation of High-frequency Components for Medical Foundation Models

Pre-training on High Definition X-ray Images: An Experimental Study

Model Balancing Helps Low-data Training and Fine-tuning

Data-Effective Learning: A Comprehensive Medical Benchmark.

Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws

Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification

Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models

Sparsity- and Hybridity-Inspired Visual Parameter-Efficient Fine-Tuning for Medical Diagnosis

Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques