Abstract:Data curation is the problem of how to collect and organize samples into a dataset that supports efficient learning. Despite the centrality of the task, little work has been devoted towards a large-scale, systematic comparison of various curation methods. In this work, we take steps towards a formal evaluation of data curation strategies and introduce SELECT, the first large-scale benchmark of curation strategies for image classification. In order to generate baseline methods for the SELECT benchmark, we create a new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K to date. Our dataset extends ImageNet with 5 new training-data shifts, each approximately the size of ImageNet-1K itself, and each assembled using a distinct curation strategy. We evaluate our data curation baselines in two ways: (i) using each training-data shift to train identical image classification models from scratch (ii) using the data itself to fit a pretrained self-supervised representation. Our findings show interesting trends, particularly pertaining to recent methods for data curation such as synthetic data generation and lookup based on CLIP embeddings. We show that although these strategies are highly competitive for certain tasks, the curation strategy used to assemble the original ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark can illuminate the path for new methods to further reduce the gap. We release our checkpoints, code, documentation, and a link to our dataset at <a class="link-external link-https" href="https://github.com/jimmyxu123/SELECT" rel="external noopener nofollow">this https URL</a>.

Embrace Sustainable AI: Dynamic Data Subset Selection for Image Classification

The Research of Adaptive Data Selection Strategy in High-Speed Image Process

Column Subset Selection for Active Learning in Image Classification

Towards Accelerated Model Training via Bayesian Data Selection

Synthetic Data for Model Selection

Problem-dependent attention and effort in neural networks with applications to image resolution and model selection

Less is more: Selecting informative and diverse subsets with balancing constraints

Dynamic classifier selection using spectral-spatial information for hyperspectral image classification

Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classification

Unlabeled data selection for active learning in image classification

AdaSelection: Accelerating Deep Learning Training through Data Subsampling

A Weighted K-Center Algorithm for Data Subset Selection

Optimal Data Selection: An Online Distributed View

Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Vision

A New Supervised Image Classification Using Data Set Partition

SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification

Image Classification with Small Datasets: Overview and Benchmark

An Active Learning Algorithm for Image Classification Based on Difficulty and Competence

Feature Selection for Improved Automatic Gender Classification

The More, the Better? Active Silencing of Non-Positive Transfer for Efficient Multi-Domain Few-Shot Classification

A CLIP-Powered Framework for Robust and Generalizable Data Selection