Abstract:As machine learning applications embrace larger data size and model complexity, practitioners turn to distributed clusters to satisfy the increasing computational and memory demands. Recently, several parallel variants of extreme learning machine (ELM) have been proposed, some of which are based on clusters. However, the limitation of computation and memory in these variants is still not well addressed when both the data and model are very large. Our goal is to build scalable ELMs with a large number of samples and hidden neurons, parallel running on clusters without computational and memory bottlenecks while having the same output results as the sequential ELM. In this paper, we propose two parallel variants of ELM, referred to as local data and model parallel ELM (LDMP-ELM) and global data and model parallel ELM (GDMP-ELM). Both variants are implemented on clusters with Message Passing Interface (MPI) environment. They both make a tradeoff between efficiency and scalability and have complementary advantages. Collectively, these two variants are called as data and model parallel ELMs (DMP-ELMs). The advantages of DMP-ELMs over existing variants are highlighted as follows: (1) They simultaneously utilize data and model parallel techniques to improve the parallelism of ELM. (2) They have better scalability to support larger data and models due to that they have addressed the memory and computational bottlenecks appearing in existing variants. Extensive experiments conducted on four large-scale datasets show that our proposed algorithms have good scalability and achieve almost ideal speedup. To the best of our knowledge, it is the first time to successfully train a large ELM model with 50,000 hidden neurons on the mnist8m dataset with 8.1 million samples and 784 features.

Distributed Extreme Learning Machine with Alternating Direction Method of Multiplier

Extreme Learning Machine with Multiple Kernels

Genetic Ensemble of Extreme Learning Machine

Dolphin Swarm Extreme Learning Machine.

Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Distributed Extreme Learning Machine for Nonlinear Learning over Network

A Survey of Distributed and Parallel Extreme Learning Machine for Big Data.

ELM ∗ : distributed extreme learning machine with MapReduce

A Maximally Split and Adaptive Relaxed Alternating Direction Method of Multipliers for Regularized Extreme Learning Machines

A Novel Relaxed ADMM with Highly Parallel Implementation for Extreme Learning Machine.

Distributed Parallel Deep Learning of Hierarchical Extreme Learning Machine for Multimode Quality Prediction with Big Process Data

Distributed and Weighted Extreme Learning Machine for Imbalanced Big Data Learning

DMP-ELMs: Data and Model Parallel Extreme Learning Machines for Large-Scale Learning Tasks

A Unified Distributed ELM Framework with Supervised, Semi-Supervised and Unsupervised Big Data Learning

A Fast Training Algorithm for Extreme Learning Machine Based on Matrix Decomposition.

Densely Connected Deep Extreme Learning Machine Algorithm

Elastic Extreme Learning Machine For Big Data Classification

Dense Incremental Extreme Learning Machine with Accelerating Amount and Proportional Integral Differential

A-ELM: Adaptive Distributed Extreme Learning Machine with MapReduce.

A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

A Parallel Multiclassification Algorithm for Big Data Using an Extreme Learning Machine