Abstract:The rapid growth in machine learning models, especially in natural language processing and computer vision, has led to challenges when running these models on hardware with limited resources. This paper introduces Superpipeline, a new framework designed to optimize the execution of large AI models on constrained hardware during both training and inference. Our approach involves dynamically managing model execution by dividing models into individual layers and efficiently transferring these layers between GPU and CPU memory. Superpipeline reduces GPU memory usage by up to 60% in our experiments while maintaining model accuracy and acceptable processing speeds. This allows models that would otherwise exceed available GPU memory to run effectively. Unlike existing solutions that focus mainly on inference or specific model types, Superpipeline can be applied to large language models (LLMs), vision-language models (VLMs), and vision-based models. We tested Superpipeline's performance across various models and hardware setups. The method includes two key parameters that allow fine-tuning the balance between GPU memory use and processing speed. Importantly, Superpipeline does not require retraining or changing model parameters, ensuring that the original model's output remains unchanged. Superpipeline's simplicity and flexibility make it useful for researchers and professionals working with advanced AI models on limited hardware. It enables the use of larger models or bigger batch sizes on existing hardware, potentially speeding up innovation across many machine learning applications. This work marks an important step toward making advanced AI models more accessible and optimizing their deployment in resource-limited environments. The code for Superpipeline is available at <a class="link-external link-https" href="https://github.com/abbasiReza/super-pipeline" rel="external noopener nofollow">this https URL</a>.

Efficient Large Models Fine-tuning on Commodity Servers Via Memory-balanced Pipeline Parallelism

Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers

Practical offloading for fine-tuning LLM on commodity GPU via learned subspace projectors

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

An Efficient 2D Method for Training Super-Large Deep Learning Models

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

DELTA: Memory-Efficient Training Via Dynamic Fine-Grained Recomputation and Swapping

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers Via Memory-Saving Inter-Operator Parallelism

A Memory-efficient Hybrid Parallel Framework for Deep Neural Network Training

Pipeline-based Optimization Method for Large-Scale End-to-End Inference.

Accelerating the Training of Large Language Models Using Efficient Activation Rematerialization and Optimal Hybrid Parallelism.

LoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPU

Improving Automatic Parallel Training Via Balanced Memory Workload Optimization

Efficient Large-Scale Language Model Training on GPU Clusters

BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training

Multi-level Storage Optimization for Intermediate Data in AI Model Training

Data-parallel distributed training of very large models beyond GPU capacity

Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training