Abstract:Training an effective Machine learning (ML) model is an iterative process that requires effort in multiple dimensions. Vertically, a single pipeline typically includes an initial ETL (Extract, Transform, Load) of raw datasets, a model training stage, and an evaluation stage where the practitioners obtain statistics of the model performance. Horizontally, many such pipelines may be required to find the best model within a search space of model configurations. Many practitioners resort to maintaining logs manually and writing simple glue code to automate the workflow. However, carrying out this process on the cloud is not a trivial task in terms of resource provisioning, data management, and bookkeeping of job histories to make sure the results are reproducible. We propose an end-to-end cloud-based machine learning platform, Accelerated Cloud for AI (ACAI), to help improve the productivity of ML practitioners. ACAI achieves this goal by enabling cloud-based storage of indexed, labeled, and searchable data, as well as automatic resource provisioning, job scheduling, and experiment tracking. Specifically, ACAI provides practitioners (1) a data lake for storing versioned datasets and their corresponding metadata, and (2) an execution engine for executing ML jobs on the cloud with automatic resource provisioning (auto-provision), logging and provenance tracking. To evaluate ACAI, we test the efficacy of our auto-provisioner on the MNIST handwritten digit classification task, and we study the usability of our system using experiments and interviews. We show that our auto-provisioner produces a 1.7x speed-up and 39% cost reduction, and our system reduces experiment time for ML scientists by 20% on typical ML use cases.

Angel: a New Large-Scale Machine Learning System

Angel+: A Large-Scale Machine Learning Platform on Angel

KunPeng: Parameter Server Based Distributed Learning Systems and Its Applications in Alibaba and Ant Financial

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

A Survey on Large-Scale Machine Learning

Parallelization of Machine Learning Algorithms Respectively on Single Machine and Spark

Petuum: A New Platform for Distributed Machine Learning on Big Data

MALT: distributed data-parallelism for existing ML applications

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Automating model search for large scale machine learning

DSANA: A Distributed Machine Learning Acceleration Solution Based on Dynamic Scheduling and Network Acceleration

HotML: A DSM-based Machine Learning System for Social Networks

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Accelerated Cloud for Artificial Intelligence (ACAI)

MLbase: A Distributed Machine-learning System.

Automatic Task Parallelization of Dataflow Graphs in ML/DL models

A Generic Online Parallel Learning Framework for Large Margin Models

BigDL 2.0: Seamless Scaling of AI Pipelines from Laptops to Distributed Cluster

MLI: An API for distributed machine learning