A hierarchical structure of extreme learning machine (HELM) for high-dimensional datasets with noise

HeYan-Lin,GengZhi-Qiang,XuYuan,ZhuQun-Xiong

IF: 6

2014-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM), a competitive machine learning technique for single-hidden-layer feedforward neural networks (SLFNNs), is simple in theory and fast in implementation. To deal with h...

What problem does this paper attempt to address?

A Hierarchical Structure of Extreme Learning Machine (HELM) for High-Dimensional Datasets with Noise

Yan-Lin He,Zhi-Qiang Geng,Yuan Xu,Qun-Xiong Zhu

DOI: https://doi.org/10.1016/j.neucom.2013.08.024

IF: 6

2014-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM), a competitive machine learning technique for single-hidden-layer feedforward neural networks (SLFNNs), is simple in theory and fast in implementation. To deal with high-dimensional data with noise, ELM with a hierarchical structure (HELM) is proposed in this paper. The proposed HELM consists of two parts: some groups of subnets and a main net. The subnets are based on some well-trained auto-associative neural networks (AANNs), which can reduce dimension and filter out noise. The main net is based on the traditional ELM. Additionally, from the perspective of data attributes spaces (DASs), the difficulties in designing subnets are avoided by using a method of Data Attributes Extension Classification (DAEC). Experiments on five high-dimensional datasets with noise are carried out to examine the HELM model. Experimental results show that HELM has higher accuracy with fewer neurons in the main net than ELM.
Genetic Ensemble of Extreme Learning Machine

Xiaowei Xue,Min Yao,Zhaohui Wu,Jianhua Yang

DOI: https://doi.org/10.1016/j.neucom.2013.09.042

IF: 6

2013-01-01

Neurocomputing

Abstract:Extreme learning machine (ELM) was proposed as a new learning algorithm to train single-hidden-layer feedforward neural networks (SLFNs). ELM has been proven to perform in high efficiency, however, due to the random determination of parameters for hidden nodes, some un-optimal parameters may be generated to influence the generalization performance and stability. Moreover, ELM may suffer from overtraining problem as the entire training dataset is used to minimize training error. In this paper, a hybrid model is proposed to alleviate such weaknesses of ELM. The model adopts genetic algorithms (GAs) to produce a group of candidate networks first, and according to a specific ranking strategy, some of the networks are selected to ensemble a new network. To verify the performance of our method, empirical comparisons were carried out with the canonical ELM, E-ELM, simple ensemble, EE-ELM, EN-ELM, Bagging and Adaboost to solve both regression and classification problems. The results have shown that our method is able to generate more robust networks with better generalization performance.
Extreme Learning Machine with Multiple Kernels

Li-juan Su,Min Yao

DOI: https://doi.org/10.1109/icca.2013.6565148

2013-01-01

Abstract:Recently a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden layer feedforward neural networks (SLFNs). Compared with other traditional gradient-descent-based learning algorithms, ELM has shown promising results because it chooses weights and biases of hidden nodes randomly and obtains the output weights and biases analytically. In most cases, ELM is fast and presents good generalization, but we find that the stability and generalization performance still can be improved. In this paper, we propose a hybrid model which combines the advantage of ELM and the advantage of Bayesian "sum of kernels" model, named Extreme Learning Machine with Multiple Kernels (MK-ELM). This method optimizes the kernel function using a weighted sum of kernel functions by a prior knowledge. Experimental results show that this approach is able to make neural networks more robust and generates better generalization performance for both regression and classification applications.
Image super-resolution representation via image patches based on extreme learning machine

qiuxi zhu,xiaodong li,weijie mao

DOI: https://doi.org/10.2991/icsecs-13.2013.61

2013-01-01

Abstract:In this paper, aimed at the extensively existing problem of slowness in mainstream image super-resolutions, an efficient approach is proposed for super-resolution based on the extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs). Features and issues (e.g. parameter selections) in the application of ELM are discussed, on the basis of which a general framework for a variety of super-resolution problems is proposed, and corresponding experiments are conducted. It is shown in the results that the proposed approach can achieve relatively good quality and much faster speed compared to traditional reconstruction-based super-resolutions, therefore the effectiveness of this method is demonstrated. Keywords-ELM; neural network; image processing; super- resolution
Hierarchical Ensemble of Extreme Learning Machine.

Yaoming Cai,Xiaobo Liu,Yongshan Zhang,Zhihua Cai

DOI: https://doi.org/10.1016/j.patrec.2018.06.015

IF: 4.757

2018-01-01

Pattern Recognition Letters

Abstract:Extreme Learning Machine (ELM), which is proposed for generalized single-hidden layer feedforward neural networks, has become a popular research topic due to its unique characteristics. However, the random nature inherent in ELM's hidden layer results in unstable performance and a large number of hidden neurons is required, making the risk of overfitting increased. In this paper, we propose a simple but effective ensemble approach, called Hierarchical Ensemble of Extreme Learning Machine (HE-ELM), to improve ELM. To encourage the diversity of component ELMs, two strategies are taken into account, namely, the sparse connection to component ELMs and feature bagging. The resulting architecture is able to integrate both representation learning and ensemble learning with relatively fewer parameters and consists of independent component ELMs, making it easy to implement, train, and apply in practice. We compare results of the proposed HE-ELM with existing methods for 22 classification problems, showing that HEELM is able to achieve significant improvement in terms of classification accuracy, with a reduced risk of overfitting the training data. (C) 2018 Elsevier B.V. All rights reserved.
Optimizing Extreme Learning Machine Via Generalized Hebbian Learning and Intrinsic Plasticity Learning

Chao Chen,Xinyu Jin,Boyuan Jiang,Lanjuan Li

DOI: https://doi.org/10.1007/s11063-018-9869-6

IF: 2.565

2018-01-01

Neural Processing Letters

Abstract:Traditional extreme learning machine (ELM) has random weights between input layer and hidden layer, this kind of random feature mapping brings non-discriminative feature space and unstable classification accuracy, which greatly limits the performance of the ELM networks. Therefore, to get the well-pleasing input weights, two biologically inspired, unsupervised learning methods were introduced to optimize the traditional ELM networks, namely the generalized hebbian algorithm (GHA) and intrinsic plasticity learning (IPL). The GHA is able to extract the principal components of the input data of arbitrary size, while the IPL tunes the probability density of the neuron’s output towards a desired distribution such as exponential distribution or weber distribution, thereby maximizing the networks information transmission. With the incorporation of the GHA and IPL approach, the optimized ELM networks generates a discriminative feature space and preserves much more characteristic of the input data, accordingly, achieving a better task performance. Based on the above two unsupervised methods, a simple, yet effective hierarchical feature mapping extreme learning machine (HFMELM) is further proposed. With almost no information loss in the layer-wise feature mapping process, the HFMELM is able to learn the high-level representation of the input data. To evaluate the effectiveness of the proposed methods, extensive experiments on several datasets are presented, the results show that the proposed methods significantly outperform the traditional ELM networks.
Extreme Learning Machines on High Dimensional and Large Data Applications: A Survey

jiuwen cao,zhiping lin

DOI: https://doi.org/10.1155/2015/103796

IF: 1.43

2015-01-01

Mathematical Problems in Engineering

Abstract:Extreme learning machine (ELM) has been developed for single hidden layer feedforward neural networks (SLFNs). In ELM algorithm, the connections between the input layer and the hidden neurons are randomly assigned and remain unchanged during the learning process. The output connections are then tuned via minimizing the cost function through a linear system. The computational burden of ELM has been significantly reduced as the only cost is solving a linear system. The low computational complexity attracted a great deal of attention from the research community, especially for high dimensional and large data applications. This paper provides an up-to-date survey on the recent developments of ELM and its applications in high dimensional and large data. Comprehensive reviews on image processing, video processing, medical signal processing, and other popular large data applications with ELM are presented in the paper.
Study on Deep Structure of Extreme Learning Machine (DS-ELM) for Datasets with Noise

meng meng ma,bo he

DOI: https://doi.org/10.4028/www.scientific.net/AMR.989-994.3679

2014-01-01

Abstract:Extreme learning machine (ELM), a relatively novel machine learning algorithm for single hidden layer feed-forward neural networks (SLFNs), has been shown competitive performance in simple structure and superior training speed. To improve the effectiveness of ELM for dealing with noisy datasets, a deep structure of ELM, short for DS-ELM, is proposed in this paper. DS-ELM contains three level networks (actually contains three nets ): the first level network is trained by auto-associative neural network (AANN) aim to filter out noise as well as reduce dimension when necessary; the second level network is another AANN net aim to fix the input weights and bias of ELM; and the last level network is ELM. Experiments on four noisy datasets are carried out to examine the new proposed DS-ELM algorithm. And the results show that DS-ELM has higher performance than ELM when dealing with noisy data.
Densely Connected Deep Extreme Learning Machine Algorithm

Jiang X. W.,Yan T. H.,Zhu J. J.,He B.,Li W. H.,Du H. P.,Sun S. S.

DOI: https://doi.org/10.1007/s12559-020-09752-2

IF: 4.89

2020-01-01

Cognitive Computation

Abstract:As a single hidden layer feed-forward neural network, the extreme learning machine (ELM) has been extensively studied for its short training time and good generalization ability. Recently, with the deep learning algorithm becoming a research hotspot, some deep extreme learning machine algorithms such as multi-layer extreme learning machine (ML-ELM) and hierarchical extreme learning machine (H-ELM) have also been proposed. However, the deep ELM algorithm also has many shortcomings: (1) when the number of model layers is shallow, the random feature mapping makes the sample features cannot be fully learned and utilized; (2) when the number of model layers is deep, the validity of the sample features will decrease after continuous abstraction and generalization. In order to solve the above problems, this paper proposes a densely connected deep ELM algorithm: dense-HELM (D-HELM). Benchmark data sets of different sizes have been employed for the property of the D-HELM algorithm. Compared with the H-ELM algorithm on the benchmark dataset, the average test accuracy is increased by 5.34% and the average training time is decreased by 21.15%. On the NORB dataset, the proposed D-HELM algorithm still maintains the best classification results and the fastest training speed. The D-HELM algorithm can make full use of the features of hidden layer learning by using the densely connected network structure and effectively reduce the number of parameters. Compared with the H-ELM algorithm, the D-HELM algorithm significantly improves the recognition accuracy and accelerates the training speed of the algorithm.
Hierarchical extreme learning machine for feedforward neural network

HanHong-Gui,WangLi-Dan,QiaoJun-Fei

IF: 6

2014-01-01

Neurocomputing

Abstract:An approach, named extended extreme learning machine (ELM), is proposed for training the weights of a class of hierarchical feedforward neural network (HFNN). Unlike conventional single-hidden-laye...
Two-stage extreme learning machine for high-dimensional data

Peng Liu,Yihua Huang,Lei Meng,Siyuan Gong,Guopeng Zhang

DOI: https://doi.org/10.1007/s13042-014-0292-7

2014-01-01

Abstract:Extreme learning machine (ELM) has been proposed for solving fast supervised learning problems by applying random computational nodes in the hidden layer. Similar to support vector machine, ELM cannot handle high-dimensional data effectively. Its generalization performance tends to become bad when it deals with high-dimensional data. In order to exploit high-dimensional data effectively, a two-stage extreme learning machine model is established. In the first stage, we incorporate ELM into the spectral regression algorithm to implement dimensionality reduction of high-dimensional data and compute the output weights. In the second stage, the decision function of standard ELM model is computed based on the low-dimensional data and the obtained output weights. This is due to the fact that two stages are all based on ELM. Thus, output weights in the second stage can be approximately replaced by those in the first stage. Consequently, the proposed method can be applicable to high-dimensional data at a fast learning speed. Experimental results show that the proposed two-stage ELM scheme tends to have better scalability and achieves outstanding generalization performance at a faster learning speed than ELM.
Affine Transformation Based Hierarchical Extreme Learning Machine

Rongzhi Ma,Jiuwen Cao,Tianlei Wang,Xiaoping Lai

DOI: https://doi.org/10.1109/ISCAS45731.2020.9181241

2020-01-01

Abstract:Recently, the signal hidden layer feedforward network (SLFN) based extreme learning machine (ELM) has been extended to a hierarchical learning framework (HELM). Although the HELM shows better generalization performance with lower computational complexity than many deep neural networks (DNNs), it is found that as the layer increases, the input distribution of each layer may move to the saturated regime of the non-linear activation function, which affects the generalization performance. However, few attentions have been paid to the data distribution normalization to address this issue. Thus, in this paper, an affine transformation (AT) inputs based activation function layer is introduced to normalize the data distribution and a novel AT based HELM (AT-HELM) is developed. The proposed AT-HELM can adapt the activation function inputs to the distribution of each layer and obtains better generalization performance. Experiments on 29 benchmark datasets are carried out to demonstrate the superiority of AT-HELM.
A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

Liu Bin,Yang Youheng,Zhao Zhibiao,Wu Chao,Liu Haoran,Wen Yan

DOI: https://doi.org/10.11999/jeit190502

2020-01-01

Abstract:As a new type of neural network, Extreme Learning Machine (ELM) has extremely fast training speed and good generalization performance. Considering the problem that the Extreme Learning Machine has high computational complexity and huge memory demand when dealing with high dimensional data, a Batch inheritance Extreme Learning Machine (B-ELM) algorithm is proposed. Firstly, the dataset is divided into different batches, and the automatic encoder network is used to reduce the dimension of each batch. Secondly, the inheritance factor is introduced to establish the relationship between adjacent batches. At the same time, the Lagrange optimization function is constructed by combining the regularization framework to realize the mathematical modeling of batch ELM. Finally, the MNIST, NORB and CIFAR-10 datasets are used for the test experiment. The experimental results show that the proposed algorithm not only has higher classification accuracy, but also reduces effectively computational complexity and memory consumption.
A study on effectiveness of extreme learning machine

Yuguang Wang,Feilong Cao,Yubo Yuan

DOI: https://doi.org/10.48550/arXiv.1409.3924

2014-09-13

Neural and Evolutionary Computing

Abstract:Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.
Extreme Learning Machine with Kernel Model Based on Deep Learning

Shifei Ding,Lili Guo,Yanlu Hou

DOI: https://doi.org/10.1007/s00521-015-2170-y

2016-01-01

Neural Computing and Applications

Abstract:Extreme learning machine (ELM) proposed by Huang et al. is a learning algorithm for single-hidden layer feedforward neural networks (SLFNs). ELM has the advantage of fast learning speed and high efficiency, so it brought into public focus. Later someone developed regularized extreme learning machine (RELM) and extreme learning machine with kernel (KELM). But they are the single-hidden layer network structure, so they have deficient in feature extraction. Deep learning (DL) is a multi-layer network structure, and it can extract the significant features by learning from a lower layer to a higher layer. As DL mostly uses the gradient descent method, it will spend too much time in the process of adjusting parameters. This paper proposed a novel model of convolutional extreme learning machine with kernel (CKELM) which was based on DL for solving problems—KELM is deficient in feature extraction, and DL spends too much time in the training process. In CKELM model, alternate convolutional layers and subsampling layers add to hidden layer of the original KELM so as to extract features and classify. The convolutional layer and subsampling layer do not use the gradient algorithm to adjust parameters because of some architectures yielded good performance with random weights. Finally, we took experiments on USPS and MNIST database. The accuracy of CKELM is higher than ELM, RELM and KELM, which proved the validity of the optimization model. To make the proposed approach more convincing, we compared with other ELM-based methods and other DL methods and also achieved satisfactory results.
A Fast Training Algorithm for Extreme Learning Machine Based on Matrix Decomposition.

Junpeng Li,Changchun Hua,Yinggan Tang,Xinping Guan

DOI: https://doi.org/10.1016/j.neucom.2015.09.067

IF: 6

2015-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM), a competitive machine learning technique for single-hidden-layer feedforward neural networks (SLFNNs), has proven to be efficient and effective algorithm for regression and classification problems. However, traditional ELM involves a large number of hidden nodes for complex real world regression and classification problems which increasing the computation burden. In this paper, a decomposition based fast ELM (DFELM) algorithm is proposed to effectively reduce the computational burden for large number of hidden nodes condition. Compared with ELM algorithm, DFELM algorithm has faster training time with a large number of hidden nodes maintaining the same accuracy performance. Experiment on three regression problems, six classification problems and a complex blast furnace modeling problem are carried out to verify the performance of DFELM algorithm. Moreover, the decomposition method can be extended to other modified ELM algorithms to further reduce the training time.
A Novel Extreme Learning Machine Based on Hybrid Kernel Function.

Shifei Ding,Yanan Zhang,Xinzheng Xu,Lina Bao

DOI: https://doi.org/10.4304/jcp.8.8.2110-2117

2013-01-01

Journal of Computers

Abstract:Extreme learning machine is a new learning algorithm for the single hidden layer feedforward neural networks (SLFNs). ELM has been widely used in various fields and applications to overcome the slow training speed and over-fitting problems of the conventional neural network learning algorithms. ELM algorithm is based on the empirical risk minimization, without considering the structural risk and this may lead to over-fitting problems and at the same time, it is with poor controllability and robustness. For these deficiencies, an optimization method is proposed in this paper, a novel extreme learning machine based on hybrid kernel function (HKELM). The method constructs a hybrid kernel function with better performance by fully combining local kernel function strong learning ability and global kernel function strong generalization ability. Compared with traditional ELM, the results show that this method can effectively improve the ELM classification results, avoid local minimum, with better generalization, robustness, controllability and faster learning rate.
Ensemble based extreme learning machine

Nan Liu, Han Wang

2010-06-21

Abstract:Extreme learning machine (ELM) was proposed as a new class of learning algorithm for single-hidden layer feedforward neural network (SLFN). To achieve good generalization performance, ELM minimizes training error on the entire training data set, therefore it might suffer from overfitting as the learning model will approximate all training samples well. In this letter, an ensemble based ELM (EN-ELM) algorithm is proposed where ensemble learning and cross-validation are embedded into the training phase so as to alleviate the overtraining problem and enhance the predictive stability. Experimental results on several benchmark databases demonstrate that EN-ELM is robust and efficient for classification.
Extreme learning machine: algorithm, theory and applications

Shifei Ding,Han Zhao,Yanan Zhang,Xinzheng Xu,Ru Nie

DOI: https://doi.org/10.1007/s10462-013-9405-z

IF: 9.588

2013-04-23

Artificial Intelligence Review

Abstract:Extreme learning machine (ELM) is a new learning algorithm for the single hidden layer feedforward neural networks. Compared with the conventional neural network learning algorithm it overcomes the slow training speed and over-fitting problems. ELM is based on empirical risk minimization theory and its learning process needs only a single iteration. The algorithm avoids multiple iterations and local minimization. It has been used in various fields and applications because of better generalization ability, robustness, and controllability and fast learning rate. In this paper, we make a review of ELM latest research progress about the algorithms, theory and applications. It first analyzes the theory and the algorithm ideas of ELM, then tracking describes the latest progress of ELM in recent years, including the model and specific applications of ELM, finally points out the research and development prospects of ELM in the future.

computer science, artificial intelligence

A hierarchical structure of extreme learning machine (HELM) for high-dimensional datasets with noise

A Hierarchical Structure of Extreme Learning Machine (HELM) for High-Dimensional Datasets with Noise

Genetic Ensemble of Extreme Learning Machine

Extreme Learning Machine with Multiple Kernels

Image super-resolution representation via image patches based on extreme learning machine

Hierarchical Ensemble of Extreme Learning Machine.

Optimizing Extreme Learning Machine Via Generalized Hebbian Learning and Intrinsic Plasticity Learning

Extreme Learning Machines on High Dimensional and Large Data Applications: A Survey

Study on Deep Structure of Extreme Learning Machine (DS-ELM) for Datasets with Noise

Densely Connected Deep Extreme Learning Machine Algorithm

Hierarchical extreme learning machine for feedforward neural network

Two-stage extreme learning machine for high-dimensional data

Affine Transformation Based Hierarchical Extreme Learning Machine

A Batch Inheritance Extreme Learning Machine Algorithm Based on Regular Optimization

A study on effectiveness of extreme learning machine

Extreme Learning Machine with Kernel Model Based on Deep Learning

A Fast Training Algorithm for Extreme Learning Machine Based on Matrix Decomposition.

A Novel Extreme Learning Machine Based on Hybrid Kernel Function.

Ensemble based extreme learning machine

Extreme learning machine: algorithm, theory and applications