An unsupervised discriminative extreme learning machine and its applications to data clustering

PengYong,ZhengWei-Long,LuBao-Liang

IF: 6

2016-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM), which was initially proposed for training single-layer feed-forward networks (SLFNs), provides us a unified efficient and effective framework for regression and mult...

What problem does this paper attempt to address?

An unsupervised discriminative extreme learning machine and its applications to data clustering.

Yong Peng,Wei-Long Zheng,Bao-Liang Lu

DOI: https://doi.org/10.1016/j.neucom.2014.11.097

IF: 6

2016-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM), which was initially proposed for training single-layer feed-forward networks (SLFNs), provides us a unified efficient and effective framework for regression and multiclass classification. Though various ELM variants were proposed in recent years, most of them focused on the supervised learning scenario while little effort was made to extend it into unsupervised learning paradigm. Therefore, it is of great significance to put ELM into learning tasks with only unlabeled data. One popular approach for mining knowledge from unlabeled data is based on the manifold assumption, which exploits the geometrical structure of data by assuming that nearby points will also be close to each other in transformation space. However, considering the manifold information only is insufficient for discriminative tasks. In this paper, we propose an improved unsupervised discriminative ELM (UDELM) model, whose main advantage is to combine the local manifold learning with global discriminative learning together. UDELM can be efficiently optimized by solving a generalized eigen-value decomposition problem. Extensive comparisons over several state-of-the-art models on clustering image and emotional EEG data demonstrate the efficacy of UDELM.
Extreme Learning Machine Based Transfer Learning for Data Classification.

Xiaodong Li,Weijie Mao,Wei Jiang

DOI: https://doi.org/10.1016/j.neucom.2015.01.096

IF: 6

2015-01-01

Neurocomputing

Abstract:The extreme learning machine (ELM) is a new method for using Single-hidden Layer Feed-forward Networks (SLFNs) with a much simpler training method. While conventional extreme learning machine are based on the training and test data which should be under the same distribution, in reality it is often desirable to learn an accurate model using only a tiny amount of new data and a large amount of old data. Transfer learning (TL) aims to solve related but different target domain problems by using plenty of labeled source domain data. When the task from one new domain comes, new domain samples are relabeled costly, and it would be a waste to discard all the old domain data. Therefore, an algorithm called TL-ELM based on the ELM algorithm is proposed, which uses a small amount of target domain tag data and a large number of source domain old data to build a high-quality classification model. The method inherits the advantages of ELM and makes up for the defects that traditional ELM cannot transfer knowledge. Experimental results indicate that the performance of the proposed methods is superior to or at least comparable with existing benchmarking methods. In addition, a novel domain adaptation kernel extreme learning machine (TL-DAKELM) based on the kernel extreme learning machine was proposed with respect to the TL-ELM. Experimental results show the effectiveness of the proposed algorithm.
Genetic Ensemble of Extreme Learning Machine

Xiaowei Xue,Min Yao,Zhaohui Wu,Jianhua Yang

DOI: https://doi.org/10.1016/j.neucom.2013.09.042

IF: 6

2013-01-01

Neurocomputing

Abstract:Extreme learning machine (ELM) was proposed as a new learning algorithm to train single-hidden-layer feedforward neural networks (SLFNs). ELM has been proven to perform in high efficiency, however, due to the random determination of parameters for hidden nodes, some un-optimal parameters may be generated to influence the generalization performance and stability. Moreover, ELM may suffer from overtraining problem as the entire training dataset is used to minimize training error. In this paper, a hybrid model is proposed to alleviate such weaknesses of ELM. The model adopts genetic algorithms (GAs) to produce a group of candidate networks first, and according to a specific ranking strategy, some of the networks are selected to ensemble a new network. To verify the performance of our method, empirical comparisons were carried out with the canonical ELM, E-ELM, simple ensemble, EE-ELM, EN-ELM, Bagging and Adaboost to solve both regression and classification problems. The results have shown that our method is able to generate more robust networks with better generalization performance.
Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Xiaodong Li,Weijie Mao,Wei Jiang

DOI: https://doi.org/10.1007/s00521-014-1709-7

2016-01-01

Neural Computing and Applications

Abstract:The extreme learning machine (ELM) is a new method for using single hidden layer feed-forward networks with a much simpler training method. While conventional kernel-based classifiers are based on a single kernel, in reality, it is often desirable to base classifiers on combinations of multiple kernels. In this paper, we propose the issue of multiple-kernel learning (MKL) for ELM by formulating it as a semi-infinite linear programming. We further extend this idea by integrating with techniques of MKL. The kernel function in this ELM formulation no longer needs to be fixed, but can be automatically learned as a combination of multiple kernels. Two formulations of multiple-kernel classifiers are proposed. The first one is based on a convex combination of the given base kernels, while the second one uses a convex combination of the so-called equivalent kernels. Empirically, the second formulation is particularly competitive. Experiments on a large number of both toy and real-world data sets (including high-magnification sampling rate image data set) show that the resultant classifier is fast and accurate and can also be easily trained by simply changing linear program.
Extreme Learning Machine with Multiple Kernels

Li-juan Su,Min Yao

DOI: https://doi.org/10.1109/icca.2013.6565148

2013-01-01

Abstract:Recently a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden layer feedforward neural networks (SLFNs). Compared with other traditional gradient-descent-based learning algorithms, ELM has shown promising results because it chooses weights and biases of hidden nodes randomly and obtains the output weights and biases analytically. In most cases, ELM is fast and presents good generalization, but we find that the stability and generalization performance still can be improved. In this paper, we propose a hybrid model which combines the advantage of ELM and the advantage of Bayesian "sum of kernels" model, named Extreme Learning Machine with Multiple Kernels (MK-ELM). This method optimizes the kernel function using a weighted sum of kernel functions by a prior knowledge. Experimental results show that this approach is able to make neural networks more robust and generates better generalization performance for both regression and classification applications.
Image super-resolution representation via image patches based on extreme learning machine

qiuxi zhu,xiaodong li,weijie mao

DOI: https://doi.org/10.2991/icsecs-13.2013.61

2013-01-01

Abstract:In this paper, aimed at the extensively existing problem of slowness in mainstream image super-resolutions, an efficient approach is proposed for super-resolution based on the extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs). Features and issues (e.g. parameter selections) in the application of ELM are discussed, on the basis of which a general framework for a variety of super-resolution problems is proposed, and corresponding experiments are conducted. It is shown in the results that the proposed approach can achieve relatively good quality and much faster speed compared to traditional reconstruction-based super-resolutions, therefore the effectiveness of this method is demonstrated. Keywords-ELM; neural network; image processing; super- resolution
Discriminative Clustering Via Extreme Learning Machine.

Gao Huang,Tianchi Liu,Yan Yang,Zhiping Lin,Shiji Song,Cheng Wu

DOI: https://doi.org/10.1016/j.neunet.2015.06.002

IF: 7.8

2015-01-01

Neural Networks

Abstract:Discriminative clustering is an unsupervised learning framework which introduces the discriminative learning rule of supervised classification into clustering. The underlying assumption is that a good partition (clustering) of the data should yield high discrimination, namely, the partitioned data can be easily classified by some classification algorithms. In this paper, we propose three discriminative clustering approaches based on Extreme Learning Machine (ELM). The first algorithm iteratively trains weighted ELM (W-ELM) classifier to gradually maximize the data discrimination. The second and third methods are both built on Fisher's Linear Discriminant Analysis (LDA); but one approach adopts alternative optimization, while the other leverages kernel k-means. We show that the proposed algorithms can be easily implemented, and yield competitive clustering accuracy on real world data sets compared to state-of-the-art clustering methods.
Unsupervised Extreme Learning Machine with Representational Features

Shifei Ding,Nan Zhang,Jian Zhang,Xinzheng Xu,Zhongzhi Shi

DOI: https://doi.org/10.1007/s13042-015-0351-8

2015-01-01

International Journal of Machine Learning and Cybernetics

Abstract:Extreme learning machine (ELM) is not only an effective classifier but also a useful cluster. Unsupervised extreme learning machine (US-ELM) gives favorable performance compared to state-of-the-art clustering algorithms. Extreme learning machine as an auto encoder (ELM-AE) can obtain principal components which represent original samples. The proposed unsupervised extreme learning machine based on embedded features of ELM-AE (US-EF-ELM) algorithm applies ELM-AE to US-ELM. US-EF-ELM regards embedded features of ELM-AE as the outputs of US-ELM hidden layer, and uses US-ELM to obtain the embedded matrix of US-ELM. US-EF-ELM can handle the multi-cluster clustering. The learning capability and computational efficiency of US-EF-ELM are as same as US-ELM. By experiments on UCI data sets, we compared US-EF-ELM k-means algorithm with k-means algorithm, spectral clustering algorithm, and US-ELM k-means algorithm in accuracy and efficiency.
Semi-supervised and unsupervised extreme learning machines.

Gao Huang,Shiji Song,Jatinder N D Gupta,Cheng Wu

DOI: https://doi.org/10.1109/TCYB.2014.2307349

2014-01-01

Abstract:Extreme learning machines (ELMs) have proven to be efficient and effective learning mechanisms for pattern classification and regression. However, ELMs are primarily applied to supervised learning problems. Only a few existing research papers have used ELMs to explore unlabeled data. In this paper, we extend ELMs for both semi-supervised and unsupervised tasks based on the manifold regularization, thus greatly expanding the applicability of ELMs. The key advantages of the proposed algorithms are as follows: 1) both the semi-supervised ELM (SS-ELM) and the unsupervised ELM (US-ELM) exhibit learning capability and computational efficiency of ELMs; 2) both algorithms naturally handle multiclass classification or multicluster clustering; and 3) both algorithms are inductive and can handle unseen data at test time directly. Moreover, it is shown in this paper that all the supervised, semi-supervised, and unsupervised ELMs can actually be put into a unified framework. This provides new perspectives for understanding the mechanism of random feature mapping, which is the key concept in ELM theory. Empirical study on a wide range of data sets demonstrates that the proposed algorithms are competitive with the state-of-the-art semi-supervised or unsupervised learning algorithms in terms of accuracy and efficiency.
Extreme Learning Machine Combining Hidden-Layer Feature Weighting and Batch Training for Classification

Chao Wu,Yu Sang,Yakun Gao

DOI: https://doi.org/10.1007/s11063-023-11358-2

IF: 2.565

2023-01-01

Neural Processing Letters

Abstract:Based on the network structure and training methods of extreme learning machines, extreme learning machine combining hidden-layer feature weighting and batch training (ELM-WB) is proposed to make full use of representation-level features for object images and human action videos classification. To solve the problem of insufficient fusion of multiple representation-level features in most classification methods, a double hidden layer structure in which the input layer and the second hidden layer are directly connected is designed. A loop training method of weighting coefficients and output weights is proposed based on the advantages of this structure. The proposed network structure and training method are combined to construct an extreme learning machine combining hidden-layer feature weighting (ELM-W), which can effectively fuse representation-level features to enhance the classification ability of representation-level features. On this basis, the principle of online sequential ELM (OS-ELM) is introduced to update the loop training formula of the two weights to reduce the memory consumption during the operation of the overall algorithm. ELM-WB is proposed by combining the loop training formula of two weight matrices with batch training. In order to test the feasibility of ELM-WB, experiments are conducted on Caltech 101, MSRC, UCF11 and UCF101 databases. Experimental results prove that the proposed ELM-WB can improve classification accuracy by fusing representation-level features. At the same time, ELM-WB can be used to perform classification tasks on databases of any size in a general-purpose computer without specific hardware.
Semi-Supervised and Unsupervised Extreme Learning Machines

Gao Huang,Shiji Song,Jatinder N.D. Gupta,Cheng Wu

DOI: https://doi.org/10.1109/tcyb.2014.2307349

IF: 11.8

2014-01-01

IEEE Transactions on Cybernetics

Abstract:Extreme learning machines (ELMs) have proven to be efficient and effective learning mechanisms for pattern classification and regression. However, ELMs are primarily applied to supervised learning problems. Only a few existing research papers have used ELMs to explore unlabeled data. In this paper, we extend ELMs for both semi-supervised and unsupervised tasks based on the manifold regularization, thus greatly expanding the applicability of ELMs. The key advantages of the proposed algorithms are as follows: 1) both the semi-supervised ELM (SS-ELM) and the unsupervised ELM (US-ELM) exhibit learning capability and computational efficiency of ELMs; 2) both algorithms naturally handle multiclass classification or multicluster clustering; and 3) both algorithms are inductive and can handle unseen data at test time directly. Moreover, it is shown in this paper that all the supervised, semi-supervised, and unsupervised ELMs can actually be put into a unified framework. This provides new perspectives for understanding the mechanism of random feature mapping, which is the key concept in ELM theory. Empirical study on a wide range of data sets demonstrates that the proposed algorithms are competitive with the state-of-the-art semi-supervised or unsupervised learning algorithms in terms of accuracy and efficiency.
A clustering method based on extreme learning machine.

Jinhong Huang,Zhu Liang Yu,Zhenghui Gu

DOI: https://doi.org/10.1016/j.neucom.2017.02.100

IF: 6

2018-01-01

Neurocomputing

Abstract:Though many successful methods have been proposed for supervised learning tasks, such as support vector machines and extreme learning machines (ELM), it is still an open problem to extend the successful supervised learning methods to unsupervised learning tasks and obtain better results. In this paper, we propose to extend the ELM to an unsupervised learning version and propose a clustering method based on ELM (CM-ELM) for both binary class and multiple class problems, which aims to find a labeling that would yield an optimal ELM classifier. In the ELM feature space, we propose to combine the Gaussian hidden nodes and sigmoid hidden nodes in the hidden layer to combine their advantages. Then we propose to adopt the alternative direction method to solve the non-convex problems in CM-ELM simply. Furthermore, in order to make the results of the non-convex problems robust and satisfactory, we propose to initial the labels with cluster ensemble methods. Experiments on the artificial and benchmark data sets show that the CM-ELM is competitive to the state-of-the-art clustering methods.
Discriminative manifold extreme learning machine and applications to image and EEG signal classification.

Yong Peng,Bao-Liang Lu

DOI: https://doi.org/10.1016/j.neucom.2015.03.118

IF: 6

2016-01-01

Neurocomputing

Abstract:Extreme learning machine (ELM) uses a non-iterative method to train single-hidden-layer feed-forward networks (SLFNs), which has been proven to be an efficient and effective learning model for both classification and regression. The main advantage of ELM lies in that the input weights as well as the hidden layer biases can be randomly generated, which contributes to the analytical solution of output weights. In this paper, we propose a discriminative manifold ELM (DMELM) by simultaneously considering the discriminative information and geometric structure of data; specifically, we exploit the discriminative information in the local neighborhood around each data point. To this end, a graph regularizer based on a newly designed graph Laplacian to characterize both properties is formulated and incorporated into the ELM objective. In DMELM, the output weights can also be obtained in analytical form. Extensive experiments are conducted on image and EEG signal classification to evaluate the effectiveness of DMELM. The results show that DMELM consistently achieves better performance than original ELM and yields promising results in comparison with several state-of-the-art algorithms, which suggests that both the discriminative as well as manifold information are beneficial to classification.
Information discriminative extreme learning machine

Deqin Yan,Yonghe Chu,Haiying Zhang,Deshan Liu

DOI: https://doi.org/10.1007/s00500-016-2372-y

2016-01-01

Abstract:Extreme learning machine (ELM) has become one of the new research hotspots in the field of pattern recognition and machine learning. However, the existing extreme learning machine algorithms cannot better use identification information of data. Aiming at solving this problem, we propose a regularized extreme learning machine (algorithm) based on discriminative information (called IELM). In order to evaluate and verify the effectiveness of the proposed method, experiments use widely used image data sets. The comparative experimental results show that the proposed algorithm in the paper can significantly improve the classification performance and generalization ability of ELM.
Laplacian Multi Layer Extreme Learning Machine

Shi-Fei DING,Nan ZHANG,Zhong-Zhi SHI

DOI: https://doi.org/10.13328/j.cnki.jos.005128

2017-01-01

Abstract:Extreme learning machine (ELM) not only is an effective classifier in supervised learning,but also can be applied on semi-supervised learning.However,semi-supervised ELM (SS-ELM) is merely a surface learning algorithm similar to Laplacian smooth twin support vector machine (Lap-STSVM).Deep learning has the advantage of approximating the complicated function and alleviating the optimization difficulty associated with deep models.Multi layer extreme learning machine (ML-ELM) has been developed according to the idea of deep learning and extreme learning machine,which stacks extreme learning machines,based auto encoder (ELM-AE) to create a multi-layer neural network.ML-ELM not only approximates the complicated function but also avoids the need to iterate during training process,exhibiting the merits of high learning efficiency.In this article,manifold regularization is introduced into the model of ML-ELM and a Laplacian ML-ELM (Lap-ML-ELM) is put forward.Furthermore,in order to solve the over fitting problem with ELM-AE,weight uncertainty is brought into ELM-AE to form a weight uncertainty ELM-AE (WU-ELM-AE) which can learn more robust features.Finally,a weight uncertainty Laplacian ML-ELM (WUL-ML-ELM) is proposed based on the above two algorithms,which stacks WU-ELM-AE to create a deep network and uses the manifold regularization framework to obtain the output weights.Lap-ML-ELM and WUL-ML-ELM are more efficient than SS-ELM in classification and do not need to spend too much time.Experimental results show that Lap-ML-ELM and WUL-ML-ELM are efficient semi-supervised learning algorithms.
Distributed Extreme Learning Machine for Nonlinear Learning over Network

Songyan Huang,Chunguang Li

DOI: https://doi.org/10.3390/e17020818

IF: 2.738

2015-01-01

Entropy

Abstract:Distributed data collection and analysis over a network are ubiquitous, especially over a wireless sensor network (WSN). To our knowledge, the data model used in most of the distributed algorithms is linear. However, in real applications, the linearity of systems is not always guaranteed. In nonlinear cases, the single hidden layer feedforward neural network (SLFN) with radial basis function (RBF) hidden neurons has the ability to approximate any continuous functions and, thus, may be used as the nonlinear learning system. However, confined by the communication cost, using the distributed version of the conventional algorithms to train the neural network directly is usually prohibited. Fortunately, based on the theorems provided in the extreme learning machine (ELM) literature, we only need to compute the output weights of the SLFN. Computing the output weights itself is a linear learning problem, although the input-output mapping of the overall SLFN is still nonlinear. Using the distributed algorithm to cooperatively compute the output weights of the SLFN, we obtain a distributed extreme learning machine (dELM) for nonlinear learning in this paper. This dELM is applied to the regression problem and classification problem to demonstrate its effectiveness and advantages.
Two-stage extreme learning machine for high-dimensional data

Peng Liu,Yihua Huang,Lei Meng,Siyuan Gong,Guopeng Zhang

DOI: https://doi.org/10.1007/s13042-014-0292-7

2014-01-01

Abstract:Extreme learning machine (ELM) has been proposed for solving fast supervised learning problems by applying random computational nodes in the hidden layer. Similar to support vector machine, ELM cannot handle high-dimensional data effectively. Its generalization performance tends to become bad when it deals with high-dimensional data. In order to exploit high-dimensional data effectively, a two-stage extreme learning machine model is established. In the first stage, we incorporate ELM into the spectral regression algorithm to implement dimensionality reduction of high-dimensional data and compute the output weights. In the second stage, the decision function of standard ELM model is computed based on the low-dimensional data and the obtained output weights. This is due to the fact that two stages are all based on ELM. Thus, output weights in the second stage can be approximately replaced by those in the first stage. Consequently, the proposed method can be applicable to high-dimensional data at a fast learning speed. Experimental results show that the proposed two-stage ELM scheme tends to have better scalability and achieves outstanding generalization performance at a faster learning speed than ELM.
Unsupervised and Semi-Supervised Extreme Learning Machine with Wavelet Kernel for High Dimensional Data

Nan Zhang,Shifei Ding

DOI: https://doi.org/10.1007/s12293-016-0198-x

2016-01-01

Memetic Computing

Abstract:Extreme learning machine (ELM) not only is an effective classifier in supervised learning, but also can be applied on unsupervised learning and semi-supervised learning. The model structure of unsupervised extreme learning machine (US-ELM) and semi-supervised extreme learning machine (SS-ELM) are same as ELM, the difference between them is the cost function. We introduce kernel function to US-ELM and propose unsupervised extreme learning machine with kernel (US-KELM). And SS-KELM has been proposed. Wavelet analysis has the characteristics of multivariate interpolation and sparse change, and Wavelet kernel functions have been widely used in support vector machine. Therefore, to realize a combination of the wavelet kernel function, US-ELM, and SS-ELM, unsupervised extreme learning machine with wavelet kernel function (US-WKELM) and semi-supervised extreme learning machine with wavelet kernel function (SS-WKELM) are proposed in this paper. The experimental results show the feasibility and validity of US-WKELM and SS-WKELM in clustering and classification.
An Overview of Extreme Learning Machine

Bohua Deng,Xinman Zhang,Weiyong Gong,Dongpeng Shang

DOI: https://doi.org/10.1109/crc.2019.00046

2019-01-01

Abstract:Extreme Learning Machine (ELM), as a new learning framework of Single Hidden Layer Feedforward Neural Network (SLFN), has become one of the hottest research directions in the field of artificial intelligence in recent years. It has been widely used in multiclass classification, human action recognition and other fields. ELM provides an efficient and unified learning framework for regression, classification, feature learning, and clustering. At the same time, ELM theories and algorithms have been improved for specific applications. This paper aims to provide a comprehensive review of existing research related to ELM. We first give an overview of the standard ELM. Then we discuss and analyze the typical variants of ELM, which are improved from different aspects, including models, strengths and weaknesses. Then we compare ELM with traditional methods in classification field and list the performance comparison between ELM and other deep network algorithms. Furthermore, the latest applications of ELM and its variants are summarized. Last but not least, we conclude the research trends of ELM variants mentioned above.
Discriminative graph regularized extreme learning machine and its application to face recognition.

Yong Peng,Suhang Wang,Xianzhong Long,Bao-Liang Lu

DOI: https://doi.org/10.1016/j.neucom.2013.12.065

IF: 6

2015-01-01

Neurocomputing

Abstract:Extreme Learning Machine (ELM) has been proposed as a new algorithm for training single hidden layer feed forward neural networks. The main merit of ELM lies in the fact that the input weights as well as hidden layer bias are randomly generated and thus the output weights can be obtained analytically, which can overcome the drawbacks incurred by gradient-based training algorithms such as local optima, improper learning rate and low learning speed. Based on the consistency property of data, which enforces similar samples to share similar properties, we propose a discriminative graph regularized Extreme Learning Machine (GELM) for further enhancing its classification performance in this paper. In the proposed GELM model, the label information of training samples are used to construct an adjacent graph and correspondingly the graph regularization term is formulated to constrain the output weights to learn similar outputs for samples from the same class. The proposed GELM model also has a closed form solution as the standard ELM and thus the output weights can be obtained efficiently. Experiments on several widely used face databases show that our proposed GELM can achieve much performance gain over standard ELM and regularized ELM. Moreover, GELM also performs well when compared with the state-of-the-art classification methods for face recognition.

An unsupervised discriminative extreme learning machine and its applications to data clustering

An unsupervised discriminative extreme learning machine and its applications to data clustering.

Extreme Learning Machine Based Transfer Learning for Data Classification.

Genetic Ensemble of Extreme Learning Machine

Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Extreme Learning Machine with Multiple Kernels

Image super-resolution representation via image patches based on extreme learning machine

Discriminative Clustering Via Extreme Learning Machine.

Unsupervised Extreme Learning Machine with Representational Features

Semi-supervised and unsupervised extreme learning machines.

Extreme Learning Machine Combining Hidden-Layer Feature Weighting and Batch Training for Classification

Semi-Supervised and Unsupervised Extreme Learning Machines

A clustering method based on extreme learning machine.

Discriminative manifold extreme learning machine and applications to image and EEG signal classification.

Information discriminative extreme learning machine

Laplacian Multi Layer Extreme Learning Machine

Distributed Extreme Learning Machine for Nonlinear Learning over Network

Two-stage extreme learning machine for high-dimensional data

Unsupervised and Semi-Supervised Extreme Learning Machine with Wavelet Kernel for High Dimensional Data

An Overview of Extreme Learning Machine

Discriminative graph regularized extreme learning machine and its application to face recognition.