Abstract:As machine learning applications embrace larger data size and model complexity, practitioners turn to distributed clusters to satisfy the increasing computational and memory demands. Recently, several parallel variants of extreme learning machine (ELM) have been proposed, some of which are based on clusters. However, the limitation of computation and memory in these variants is still not well addressed when both the data and model are very large. Our goal is to build scalable ELMs with a large number of samples and hidden neurons, parallel running on clusters without computational and memory bottlenecks while having the same output results as the sequential ELM. In this paper, we propose two parallel variants of ELM, referred to as local data and model parallel ELM (LDMP-ELM) and global data and model parallel ELM (GDMP-ELM). Both variants are implemented on clusters with Message Passing Interface (MPI) environment. They both make a tradeoff between efficiency and scalability and have complementary advantages. Collectively, these two variants are called as data and model parallel ELMs (DMP-ELMs). The advantages of DMP-ELMs over existing variants are highlighted as follows: (1) They simultaneously utilize data and model parallel techniques to improve the parallelism of ELM. (2) They have better scalability to support larger data and models due to that they have addressed the memory and computational bottlenecks appearing in existing variants. Extensive experiments conducted on four large-scale datasets show that our proposed algorithms have good scalability and achieve almost ideal speedup. To the best of our knowledge, it is the first time to successfully train a large ELM model with 50,000 hidden neurons on the mnist8m dataset with 8.1 million samples and 784 features.

A Parallel Multiclassification Algorithm for Big Data Using an Extreme Learning Machine

Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Multi-class AdaBoost ELM

Extreme Learning Machine and Its Applications in Big Data Processing

MR-ELM: a MapReduce-based framework for large-scale ELM training in big data era

A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

ELM ∗ : distributed extreme learning machine with MapReduce

Elastic Extreme Learning Machine For Big Data Classification

A Survey of Distributed and Parallel Extreme Learning Machine for Big Data.

Study of ELM Algorithm Parallelization Based on Spark

ELM-MapReduce: MapReduce Accelerated Extreme Learning Machine for Big Spatial Data Analysis

Generalized eigenvalue extreme learning machine for classification

An Integration of Extreme Learning Machine for Classification of Big Data

Parallel Ensemble of Online Sequential Extreme Learning Machine Based on MapReduce

Distributed Parallel Deep Learning of Hierarchical Extreme Learning Machine for Multimode Quality Prediction with Big Process Data

DMP-ELMs: Data and Model Parallel Extreme Learning Machines for Large-Scale Learning Tasks

Extreme Learning Machine Ensemble Classifier for Large-Scale Data

Efficient Batch Parallel Online Sequential Extreme Learning Machine Algorithm Based On Mapreduce

Extreme Learning Machine for Large-Scale Graph Classification Based on MapReduce

PR-ELM: Parallel regularized extreme learning machine based on cluster.

A Fast Training Algorithm for Extreme Learning Machine Based on Matrix Decomposition.