Abstract:The rapid advancements in Large Vision Models (LVMs), such as Vision Transformers (ViTs) and diffusion models, have led to an increasing demand for computational resources, resulting in substantial financial and environmental costs. This growing challenge highlights the necessity of developing efficient training methods for LVMs. Progressive learning, a training strategy in which model capacity gradually increases during training, has shown potential in addressing these challenges. In this paper, we present an advanced automated progressive learning (AutoProg) framework for efficient training of LVMs. We begin by focusing on the pre-training of LVMs, using ViTs as a case study, and propose AutoProg-One, an AutoProg scheme featuring momentum growth (MoGrow) and a one-shot growth schedule search. Beyond pre-training, we extend our approach to tackle transfer learning and fine-tuning of LVMs. We expand the scope of AutoProg to cover a wider range of LVMs, including diffusion models. First, we introduce AutoProg-Zero, by enhancing the AutoProg framework with a novel zero-shot unfreezing schedule search, eliminating the need for one-shot supernet training. Second, we introduce a novel Unique Stage Identifier (SID) scheme to bridge the gap during network growth. These innovations, integrated with the core principles of AutoProg, offer a comprehensive solution for efficient training across various LVM scenarios. Extensive experiments show that AutoProg accelerates ViT pre-training by up to 1.85x on ImageNet and accelerates fine-tuning of diffusion models by up to 2.86x, with comparable or even higher performance. This work provides a robust and scalable approach to efficient training of LVMs, with potential applications in a wide range of vision tasks. Code: <a class="link-external link-https" href="https://github.com/changlin31/AutoProg-Zero" rel="external noopener nofollow">this https URL</a>

Progressive Recurrent Learning for Visual Recognition.

RECL: Responsive Resource-Efficient Continuous Learning for Video Analytics

Generalized Coarse-to-Fine Visual Recognition with Progressive Training

Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

Progressive Neural Networks for Continuous Classification of Retinal Optical Coherence Tomography Images

RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability

Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Efficient Training of Large Vision Models via Advanced Automated Progressive Learning

Automated Progressive Learning for Efficient Training of Vision Transformers

Recurrent issues with deep neural networks of visual recognition

Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks

Visual Pretraining via Contrastive Predictive Model for Pixel-Based Reinforcement Learning

Progressive Representation Learning for Real-Time UAV Tracking

RLPeri: Accelerating Visual Perimetry Test with Reinforcement Learning and Convolutional Feature Extraction

EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones.

Proper Reuse of Features Extractor for Real-time Continual Learning

Progressive Learning for Neuronal Population Reconstruction from Optical Microscopy Images

Joint Task-Recursive Learning for RGB-D Scene Understanding

Progressive Learning Vision Transformer for Open Set Recognition of Fine-Grained Objects in Remote Sensing Images.

Progressive Visual Prompt Learning with Contrastive Feature Re-formation

Progressive Multi-Stage Learning for Discriminative Tracking