Parallelized extreme learning machine ensemble based on min-max modular network

WangXiao-Lin,ChenYang-Yang,ZhaoHai,LuBao-Liang

IF: 6

2014-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM) as an emergent technology has shown its promising performance in many applications. This paper proposes a parallelized ELM ensemble based on the Min-Max Modular netwo...

What problem does this paper attempt to address?

Genetic Ensemble of Extreme Learning Machine

Xiaowei Xue,Min Yao,Zhaohui Wu,Jianhua Yang

DOI: https://doi.org/10.1016/j.neucom.2013.09.042

IF: 6

2013-01-01

Neurocomputing

Abstract:Extreme learning machine (ELM) was proposed as a new learning algorithm to train single-hidden-layer feedforward neural networks (SLFNs). ELM has been proven to perform in high efficiency, however, due to the random determination of parameters for hidden nodes, some un-optimal parameters may be generated to influence the generalization performance and stability. Moreover, ELM may suffer from overtraining problem as the entire training dataset is used to minimize training error. In this paper, a hybrid model is proposed to alleviate such weaknesses of ELM. The model adopts genetic algorithms (GAs) to produce a group of candidate networks first, and according to a specific ranking strategy, some of the networks are selected to ensemble a new network. To verify the performance of our method, empirical comparisons were carried out with the canonical ELM, E-ELM, simple ensemble, EE-ELM, EN-ELM, Bagging and Adaboost to solve both regression and classification problems. The results have shown that our method is able to generate more robust networks with better generalization performance.
Parallelized extreme learning machine ensemble based on min-max modular network

Xiao-Lin Wang,Yang-Yang Chen,Hai Zhao,Bao-Liang Lu

DOI: https://doi.org/10.1016/j.neucom.2013.02.053

IF: 6

2014-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM) as an emergent technology has shown its promising performance in many applications. This paper proposes a parallelized ELM ensemble based on the Min-Max Modular network (M^3-network) to meet the challenge of the so-called big data. The proposed M^3-ELM first decomposes classification problems into smaller subproblems, then trains an ELM for each subproblem, and in the end ensembles these ELMs with the M^3-network. Twelve data sets including both benchmarks and real-world applications are employed to test the proposed method. The experimental results show that M^3-ELM not only speeds up the training phrases by 1.6-4.6 times but also reduces the test errors by 0.37-19.51% compared with the normal ELM. The results also indicate that M^3-ELM possesses scalability on large-scale tasks and accuracy improvement on imbalanced tasks.
Extreme Learning Machine with Multiple Kernels

Li-juan Su,Min Yao

DOI: https://doi.org/10.1109/icca.2013.6565148

2013-01-01

Abstract:Recently a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden layer feedforward neural networks (SLFNs). Compared with other traditional gradient-descent-based learning algorithms, ELM has shown promising results because it chooses weights and biases of hidden nodes randomly and obtains the output weights and biases analytically. In most cases, ELM is fast and presents good generalization, but we find that the stability and generalization performance still can be improved. In this paper, we propose a hybrid model which combines the advantage of ELM and the advantage of Bayesian "sum of kernels" model, named Extreme Learning Machine with Multiple Kernels (MK-ELM). This method optimizes the kernel function using a weighted sum of kernel functions by a prior knowledge. Experimental results show that this approach is able to make neural networks more robust and generates better generalization performance for both regression and classification applications.
Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Xiaodong Li,Weijie Mao,Wei Jiang

DOI: https://doi.org/10.1007/s00521-014-1709-7

2016-01-01

Neural Computing and Applications

Abstract:The extreme learning machine (ELM) is a new method for using single hidden layer feed-forward networks with a much simpler training method. While conventional kernel-based classifiers are based on a single kernel, in reality, it is often desirable to base classifiers on combinations of multiple kernels. In this paper, we propose the issue of multiple-kernel learning (MKL) for ELM by formulating it as a semi-infinite linear programming. We further extend this idea by integrating with techniques of MKL. The kernel function in this ELM formulation no longer needs to be fixed, but can be automatically learned as a combination of multiple kernels. Two formulations of multiple-kernel classifiers are proposed. The first one is based on a convex combination of the given base kernels, while the second one uses a convex combination of the so-called equivalent kernels. Empirically, the second formulation is particularly competitive. Experiments on a large number of both toy and real-world data sets (including high-magnification sampling rate image data set) show that the resultant classifier is fast and accurate and can also be easily trained by simply changing linear program.
A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

Liu Bin,Yang Youheng,Zhao Zhibiao,Wu Chao,Liu Haoran,Wen Yan

DOI: https://doi.org/10.11999/jeit190502

2020-01-01

Abstract:As a new type of neural network, Extreme Learning Machine (ELM) has extremely fast training speed and good generalization performance. Considering the problem that the Extreme Learning Machine has high computational complexity and huge memory demand when dealing with high dimensional data, a Batch inheritance Extreme Learning Machine (B-ELM) algorithm is proposed. Firstly, the dataset is divided into different batches, and the automatic encoder network is used to reduce the dimension of each batch. Secondly, the inheritance factor is introduced to establish the relationship between adjacent batches. At the same time, the Lagrange optimization function is constructed by combining the regularization framework to realize the mathematical modeling of batch ELM. Finally, the MNIST, NORB and CIFAR-10 datasets are used for the test experiment. The experimental results show that the proposed algorithm not only has higher classification accuracy, but also reduces effectively computational complexity and memory consumption.
Evolutionary extreme learning machine ensembles with size control

Dianhui Wang,Monther Alhamdoosh

DOI: https://doi.org/10.1016/j.neucom.2011.12.046

IF: 6

2013-01-01

Neurocomputing

Abstract:Ensemble learning aims to improve the generalization power and the reliability of learner models through sampling and optimization techniques. It has been shown that an ensemble constructed by a selective collection of base learners outperforms favorably. However, effective implementation of such an ensemble from a given learner pool is still an open problem. This paper presents an evolutionary approach for constituting extreme learning machine (ELM) ensembles. Our proposed algorithm employs the model diversity as fitness function to direct the selection of base learners, and produces an optimal solution with ensemble size control. A comprehensive comparison is carried out, where the basic ELM is used to generate a set of neural networks and 12 benchmarked regression datasets are employed in simulations. Our reporting results demonstrate that the proposed method outperforms other ensembling techniques, including simple average, bagging and adaboost, in terms of both effectiveness and efficiency.
Emergent on-line learning in min-max modular neural networks

Bao-Liang Lu,Ichikawa, M.

DOI: https://doi.org/10.1109/IJCNN.2001.938788

2001-01-01

Abstract:This paper presents a novel online supervised learning model called emergent online learning for pattern classification. The model involves three mechanisms: decomposition of an online learning problem at each time step into a reasonable number of linearly separable problems; parallel learning of these linearly separable problems by using linear threshold gates; and integration of the trained linear threshold gates into a min-max modular network. Two simple emergent laws are used to control both the problem decomposition and solution integration. The advantages of the model are very fast learning speed, guaranteed convergence, high modularity, and parallelism
Massively Parallel Classification of Single-Trial EEG Signals Using a Min-Max Modular Neural Network

Bao-Liang Lu,Jonghan Shin,Michinori Ichikawa

DOI: https://doi.org/10.1109/tbme.2003.821023

2001-01-01

Abstract:This paper presents a method for classifying single-trial electroencephalogram (EEG) signals using min-max modular neural networks implemented in a massively parallel way. The method has three main steps. First, a large-scale, complex EEG classification problem is simply divided into a reasonable number of two-class subproblems, as small as needed. Second, the two-class subproblems are simply learned by individual smaller network modules in parallel. Finally, all the individual trained network modules are integrated into a hierarchical, parallel, and modular classifier according to two module combination laws. To demonstrate the effectiveness of the method, we perform simulations on fifteen different four-class EEG classification tasks, each of which consists of 1491 training and 636 test data. These EEG classification tasks were created using a set of non-averaged, single-trial hippocampal EEG signals recorded from rats; the features of the EEG signals are extracted using wavelet transform techniques. The experimental results indicate that the proposed method has several attractive features. 1) The method is appreciably faster than the existing approach that is based on conventional multilayer perceptrons. 2) Complete learning of complex, EEG classification problems can be easily realized, and better generalization performance can be achieved. 3) The method scales up to large-scale, complex EEG classification problems.
Optimizing Extreme Learning Machine Via Generalized Hebbian Learning and Intrinsic Plasticity Learning

Chao Chen,Xinyu Jin,Boyuan Jiang,Lanjuan Li

DOI: https://doi.org/10.1007/s11063-018-9869-6

IF: 2.565

2018-01-01

Neural Processing Letters

Abstract:Traditional extreme learning machine (ELM) has random weights between input layer and hidden layer, this kind of random feature mapping brings non-discriminative feature space and unstable classification accuracy, which greatly limits the performance of the ELM networks. Therefore, to get the well-pleasing input weights, two biologically inspired, unsupervised learning methods were introduced to optimize the traditional ELM networks, namely the generalized hebbian algorithm (GHA) and intrinsic plasticity learning (IPL). The GHA is able to extract the principal components of the input data of arbitrary size, while the IPL tunes the probability density of the neuron’s output towards a desired distribution such as exponential distribution or weber distribution, thereby maximizing the networks information transmission. With the incorporation of the GHA and IPL approach, the optimized ELM networks generates a discriminative feature space and preserves much more characteristic of the input data, accordingly, achieving a better task performance. Based on the above two unsupervised methods, a simple, yet effective hierarchical feature mapping extreme learning machine (HFMELM) is further proposed. With almost no information loss in the layer-wise feature mapping process, the HFMELM is able to learn the high-level representation of the input data. To evaluate the effectiveness of the proposed methods, extensive experiments on several datasets are presented, the results show that the proposed methods significantly outperform the traditional ELM networks.
A Novel Bio-inspired Image Recognition Network with Extreme Learning Machine

lin zhang,yu zhang,ping li

DOI: https://doi.org/10.1007/978-3-319-14063-6_12

2015-01-01

Abstract:This paper presents a novel bio-inspired network for image recognition. The HMAX model and the extreme learning machine (ELM) are combined, to construct a five-layer feed-forward network: S1-C1-S2-C2-H. The previous four layers, originating from HMAX, provide robust feature representation of specific object, and the feature classification stage at the H layer is implemented with ELM. The HMAX model simulates the hierarchical processing mechanism in primate visual cortex, to calculate feature representation. As a biological learning algorithm for SLFNs, ELM learns much faster with good generalization, and performs well in classification applications. Our experimental results show effective accuracy performance with fast learning speed.
Robust Matching Pursuit Extreme Learning Machines

Zejian Yuan,Xin Wang,Jiuwen Cao,Haiquan Zhao,Badong Chen

DOI: https://doi.org/10.1155/2018/4563040

2018-01-01

Scientific Programming

Abstract:Extreme learning machine (ELM) is a popular learning algorithm for single hidden layer feedforward networks (SLFNs). It was originally proposed with the inspiration from biological learning and has attracted massive attentions due to its adaptability to various tasks with a fast learning ability and efficient computation cost. As an effective sparse representation method, orthogonal matching pursuit (OMP) method can be embedded into ELM to overcome the singularity problem and improve the stability. Usually OMP recovers a sparse vector by minimizing a least squares (LS) loss, which is efficient for Gaussian distributed data, but may suffer performance deterioration in presence of non-Gaussian data. To address this problem, a robust matching pursuit method based on a novel kernel risk-sensitive loss (in short KRSLMP) is first proposed in this paper. The KRSLMP is then applied to ELM to solve the sparse output weight vector, and the new method named the KRSLMP-ELM is developed for SLFN learning. Experimental results on synthetic and real-world data sets confirm the effectiveness and superiority of the proposed method.
An Extreme Learning Machine Based on Artificial Immune System.

Hui-yuan Tian,Shi-jian Li,Tianqi Wu,Min Yao

DOI: https://doi.org/10.1155/2018/3635845

IF: 3.12

2018-01-01

Computational Intelligence and Neuroscience

Abstract:Extreme learning machine algorithm proposed in recent years has been widely used in many fields due to its fast training speed and good generalization performance. Unlike the traditional neural network, the ELM algorithm greatly improves the training speed by randomly generating the relevant parameters of the input layer and the hidden layer. However, due to the randomly generated parameters, some generated "bad" parameters may be introduced to bring negative effect on the final generalization ability. To overcome such drawback, this paper combines the artificial immune system (AIS) with ELM, namely, AIS-ELM. With the help of AIS's global search and good convergence, the randomly generated parameters of ELM are optimized effectively and efficiently to achieve a better generalization performance. To evaluate the performance of AIS-ELM, this paper compares it with relevant algorithms on several benchmark datasets. The experimental results reveal that our proposed algorithm can always achieve superior performance.
Extreme learning machine: algorithm, theory and applications

Shifei Ding,Han Zhao,Yanan Zhang,Xinzheng Xu,Ru Nie

DOI: https://doi.org/10.1007/s10462-013-9405-z

IF: 9.588

2013-04-23

Artificial Intelligence Review

Abstract:Extreme learning machine (ELM) is a new learning algorithm for the single hidden layer feedforward neural networks. Compared with the conventional neural network learning algorithm it overcomes the slow training speed and over-fitting problems. ELM is based on empirical risk minimization theory and its learning process needs only a single iteration. The algorithm avoids multiple iterations and local minimization. It has been used in various fields and applications because of better generalization ability, robustness, and controllability and fast learning rate. In this paper, we make a review of ELM latest research progress about the algorithms, theory and applications. It first analyzes the theory and the algorithm ideas of ELM, then tracking describes the latest progress of ELM in recent years, including the model and specific applications of ELM, finally points out the research and development prospects of ELM in the future.

computer science, artificial intelligence
Distributed Extreme Learning Machine for Nonlinear Learning over Network

Songyan Huang,Chunguang Li

DOI: https://doi.org/10.3390/e17020818

IF: 2.738

2015-01-01

Entropy

Abstract:Distributed data collection and analysis over a network are ubiquitous, especially over a wireless sensor network (WSN). To our knowledge, the data model used in most of the distributed algorithms is linear. However, in real applications, the linearity of systems is not always guaranteed. In nonlinear cases, the single hidden layer feedforward neural network (SLFN) with radial basis function (RBF) hidden neurons has the ability to approximate any continuous functions and, thus, may be used as the nonlinear learning system. However, confined by the communication cost, using the distributed version of the conventional algorithms to train the neural network directly is usually prohibited. Fortunately, based on the theorems provided in the extreme learning machine (ELM) literature, we only need to compute the output weights of the SLFN. Computing the output weights itself is a linear learning problem, although the input-output mapping of the overall SLFN is still nonlinear. Using the distributed algorithm to cooperatively compute the output weights of the SLFN, we obtain a distributed extreme learning machine (dELM) for nonlinear learning in this paper. This dELM is applied to the regression problem and classification problem to demonstrate its effectiveness and advantages.
Differential Evolution Based Selective Ensemble Of Extreme Learning Machine

Yong Zhang,Bo Liu,Fan Yang

DOI: https://doi.org/10.1109/TrustCom.2016.0211

2016-01-01

Abstract:Extreme learning machine (ELM) for single-hidden-layer feedforward neural networks has been widely used in classification and regression for its fast learning speed. However, a single ELM suffers from problems of stability and overfitting. Ensemble approach can effectively resolve these problems. This paper proposes a selective ensemble learning algorithm based on differential evolution (DE) for classification problem. In the proposed algorithm, ELM is selected as base classifier, and then DE algorithm is employed as the optimization technique to construct an ensemble learning model by combining base classifiers. The weights of each base classifier in the ensemble are optimized by DE algorithm. Finally, several base classifiers with larger weights are selected to form the ensemble for making decision. Experimental results on 14 benchmark datasets demonstrate that the proposed algorithm can effectively improve the classification accuracy and generalization ability.
Multi-parallel Extreme Learning Machine with Excitatory and Inhibitory Neurons for Regression.

Guoqiang Li,Junnan Zou

DOI: https://doi.org/10.1007/s11063-019-10160-3

IF: 2.565

2019-01-01

Neural Processing Letters

Abstract:Compared with traditional neural networks, extreme learning machine (ELM) shows outstanding performances on speed and computation. Aiming at the problems that ELM needs more hidden layer neurons and meaningful features of data sometimes are sacrificed in order to improve the training speed, a novelty network multi-parallel extreme learning machine with excitatory and inhibitory neurons (MEI-ELM) is proposed based on the idea of biological neurons. In MEI-ELM, (1) A parallel system is introduced to make it more compact and reduce the number of hidden layer neurons. (2) The property of excitatory and inhibitory of biological neuronal for data processing is introduced to improve its performance. Through applying MEI-ELM, ELM, Fast Learning Network (FLN) and Fast Learning Network with Parallel Layer Perceptrons (PLP-FLN) to 11 classical regression problems, it can be obtained that MEI-ELM performs much better than the other methods in generalization and stability.
Distributed Extreme Learning Machine with Alternating Direction Method of Multiplier

Minnan Luo,Qinghua Zheng,Jun Liu

DOI: https://doi.org/10.1016/j.neucom.2016.03.112

2016-01-01

Abstract:Extreme learning machine, as a generalized single-hidden-layer feedforward networks has achieved much attention for its extremely fast learning speed and good generalization performance. However, big data often makes a challenge in large scale learning of ELM due to the limitation of memory of single machine as well as the distributed manner of large scale data storage and collection in many applications. For the purpose of relieving the limitation of memory with big data, in this paper, we exploit a novel distributed extreme learning machine to implement the extreme learning machine algorithm in parallel for large-scale data set. A corresponding distributed algorithm is also developed on the basis of alternating direction method of multipliers which shows effectiveness in distributed convex optimization. Finally, some numerical experiments on well-known benchmark data sets are carried out to illustrate the effectiveness of the proposed DELM method and provide an analysis on the performance of speedup, scaleup and sizeup.
A Novel Extreme Learning Machine Based on Hybrid Kernel Function.

Shifei Ding,Yanan Zhang,Xinzheng Xu,Lina Bao

DOI: https://doi.org/10.4304/jcp.8.8.2110-2117

2013-01-01

Journal of Computers

Abstract:Extreme learning machine is a new learning algorithm for the single hidden layer feedforward neural networks (SLFNs). ELM has been widely used in various fields and applications to overcome the slow training speed and over-fitting problems of the conventional neural network learning algorithms. ELM algorithm is based on the empirical risk minimization, without considering the structural risk and this may lead to over-fitting problems and at the same time, it is with poor controllability and robustness. For these deficiencies, an optimization method is proposed in this paper, a novel extreme learning machine based on hybrid kernel function (HKELM). The method constructs a hybrid kernel function with better performance by fully combining local kernel function strong learning ability and global kernel function strong generalization ability. Compared with traditional ELM, the results show that this method can effectively improve the ELM classification results, avoid local minimum, with better generalization, robustness, controllability and faster learning rate.
Decomposition and Parallel Learning of Imbalanced Classification Problems by Min-Max Modular Neural Network

BL Lu,M Ito

1998-01-01

Abstract:We present a method for decomposing imbalanced two-class classification problems into a set of small subproblems based on the class relations among training data. An attractive feature of the method is that any large imbalanced problem can be decomposed into a set of subproblems as small as the user expects, even though the user is not a domain specialist or the user has no any prior knowledge concerning the problem. We also present an approach to learning the subproblems in parallel by the min-max modular neural network. We show that the approach has several advantages over single networks such as easily designing of the network modules, faster learning, and good generalization performance.
An accelerated optimization algorithm for the elastic-net extreme learning machine

Zhang, Yuao,Dai, Yunwei,Wu, Qingbiao

DOI: https://doi.org/10.1007/s13042-022-01636-1

2022-09-09

International Journal of Machine Learning and Cybernetics

Abstract:Extreme learning machine (ELM) has received considerable attention due to its rapid learning speed and powerful fitting capabilities. One of its important variants, the elastic-net ELM (Enet-ELM), was recently proposed to improve its sparsity and stability of simulations simultaneously. However, entering the era of big data, the explosive growth of data volume and dimensions poses a huge challenge to Enet-ELM. On the other hand, the alternating direction method of multipliers (ADMM) is a powerful iterative algorithm for solving large-scale optimization problems by splitting a large problem into a set of executable sub-problems. But its performance is highly restricted by its astringency and convergence rates. In this paper, we therefore develop a novel Enet-ELM algorithm based on the over-relaxed ADMM, termed over-relaxed Enet-ELM (OE-ELM), which accelerates model training by applying the results of the previous iteration to the next iteration. Besides, we also propose a parallel version of OE-ELM (POE-ELM) to implement parallel and distributed computation, which is trained by the consensus over relaxation ADMM algorithm. Finally, the convergence analysis conducted on the two proposed algorithms proves the effectiveness of model training, and extensive experiments on classification and regression datasets demonstrate their competitiveness in accuracy and convergence rate.

computer science, artificial intelligence

Parallelized extreme learning machine ensemble based on min-max modular network

Genetic Ensemble of Extreme Learning Machine

Parallelized extreme learning machine ensemble based on min-max modular network

Extreme Learning Machine with Multiple Kernels

Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

Evolutionary extreme learning machine ensembles with size control

Emergent on-line learning in min-max modular neural networks

Massively Parallel Classification of Single-Trial EEG Signals Using a Min-Max Modular Neural Network

Optimizing Extreme Learning Machine Via Generalized Hebbian Learning and Intrinsic Plasticity Learning

A Novel Bio-inspired Image Recognition Network with Extreme Learning Machine

Robust Matching Pursuit Extreme Learning Machines

An Extreme Learning Machine Based on Artificial Immune System.

Extreme learning machine: algorithm, theory and applications

Distributed Extreme Learning Machine for Nonlinear Learning over Network

Differential Evolution Based Selective Ensemble Of Extreme Learning Machine

Multi-parallel Extreme Learning Machine with Excitatory and Inhibitory Neurons for Regression.

Distributed Extreme Learning Machine with Alternating Direction Method of Multiplier

A Novel Extreme Learning Machine Based on Hybrid Kernel Function.

Decomposition and Parallel Learning of Imbalanced Classification Problems by Min-Max Modular Neural Network

An accelerated optimization algorithm for the elastic-net extreme learning machine