L1-norm-based Kernel Entropy Components

Chengzu Bai,Ren Zhang,Zeshui Xu,Rui Cheng,Baogang Jin,Jian Chen

DOI: https://doi.org/10.1016/j.patcog.2019.106990

IF: 8

2019-01-01

Pattern Recognition

Abstract:•L1-KECA can achieve desirable performance on kernel density estimation.•L1-KECA can obtain acceptable results on classification and face recognition.•L1-KECA has much faster convergence than optimized KECA.

What problem does this paper attempt to address?

Optimized Kernel Entropy Components

Emma Izquierdo-Verdiguier,Valero Laparra,Robert Jenssen,Luis Gómez-Chova,Gustau Camps-Valls

DOI: https://doi.org/10.1109/TNNLS.2016.2530403

2016-03-09

Abstract:This work addresses two main issues of the standard Kernel Entropy Component Analysis (KECA) algorithm: the optimization of the kernel decomposition and the optimization of the Gaussian kernel parameter. KECA roughly reduces to a sorting of the importance of kernel eigenvectors by entropy instead of by variance as in Kernel Principal Components Analysis. In this work, we propose an extension of the KECA method, named Optimized KECA (OKECA), that directly extracts the optimal features retaining most of the data entropy by means of compacting the information in very few features (often in just one or two). The proposed method produces features which have higher expressive power. In particular, it is based on the Independent Component Analysis (ICA) framework, and introduces an extra rotation to the eigen-decomposition, which is optimized via gradient ascent search. This maximum entropy preservation suggests that OKECA features are more efficient than KECA features for density estimation. In addition, a critical issue in both methods is the selection of the kernel parameter since it critically affects the resulting performance. Here we analyze the most common kernel length-scale selection criteria. Results of both methods are illustrated in different synthetic and real problems. Results show that 1) OKECA returns projections with more expressive power than KECA, 2) the most successful rule for estimating the kernel parameter is based on maximum likelihood, and 3) OKECA is more robust to the selection of the length-scale parameter in kernel density estimation.

Machine Learning
Improved KECA for Process Monitoring Using Ensemble Learning and Bayesian Inference

Jiaxiang Qin,Chunjie Yang,Mengyuan Sun,Wenhui Liu

DOI: https://doi.org/10.1109/chicc.2016.7554405

2016-01-01

Abstract:Kernel entropy component analysis (KECA) has been introduced into process monitoring recently. Generally, there exist different kinds of faults in actually industrial process and the fault information is unknown. These faults contain different information, that a single kernel function with explicit width parameter is not effective enough to detect different types of faults. The width parameter, meanwhile, is usually determined by experience in kernel function. To address theses issues, we incorporate ensemble learning approach with Bayesian inference into KECA approach. This proposed approach overcomes the sightless of the parameter selection and is suitable for detecting different kinds of faults. We test it in the complicated Tennesse Eastman (TE) benchmark process, the result shows that the monitoring performance is markedly improved.
Sparse Kernel Entropy Component Analysis for Dimensionality Reduction of Biomedical Data

Jun Shi,Qikun Jiang,Qi Zhang,Qinghua Huang,Xuelong Li

DOI: https://doi.org/10.1016/j.neucom.2015.05.032

IF: 6

2015-01-01

Neurocomputing

Abstract:Dimensionality reduction is ubiquitous in biomedical applications. A newly proposed spectral dimensionality reduction method, named kernel entropy component analysis (KECA), can reveal the structure related to Renyi entropy of an input space data set. However, each principal component in the Hilbert space depends on all training samples in KECA, causing degraded performance. To overcome this drawback, a sparse KECA (SKECA) algorithm based on a recursive divide-and-conquer (DC) method is proposed in this work. The original large and complex problem of KECA is decomposed into a series of small and simple sub-problems, and then they are solved recursively. The performance of SKECA is evaluated on four biomedical datasets, and compared with KECA, principal component analysis (PCA), kernel PCA (KPCA), sparse PCA and sparse KPCA. Experimental results indicate that the SKECA outperforms conventional dimensionality reduction algorithms, even for high order dimensional features. It suggests that SKECA is potentially applicable to biomedical data processing.
L1-Norm Kernel Discriminant Analysis Via Bayes Error Bound Optimization for Robust Feature Extraction

Wenming Zheng,Zhouchen Lin,Haixian Wang

DOI: https://doi.org/10.1109/tnnls.2013.2281428

IF: 14.255

2014-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:A novel discriminant analysis criterion is derived in this paper under the theoretical framework of Bayes optimality. In contrast to the conventional Fisher's discriminant criterion, the major novelty of the proposed one is the use of L1 norm rather than L2 norm, which makes it less sensitive to the outliers. With the L1-norm discriminant criterion, we propose a new linear discriminant analysis (L1-LDA) method for linear feature extraction problem. To solve the L1-LDA optimization problem, we propose an efficient iterative algorithm, in which a novel surrogate convex function is introduced such that the optimization problem in each iteration is to simply solve a convex programming problem and a close-form solution is guaranteed to this problem. Moreover, we also generalize the L1-LDA method to deal with the nonlinear robust feature extraction problems via the use of kernel trick, and hereafter proposed the L1-norm kernel discriminant analysis (L1-KDA) method. Extensive experiments on simulated and real data sets are conducted to evaluate the effectiveness of the proposed method in comparing with the state-of-the-art methods.
Human Face Recognition using Gabor based Kernel Entropy Component Analysis

Arindam Kar,Debotosh Bhattacharjee,Dipak Kumar Basu,Mita Nasipuri,Mahantapas Kundu

DOI: https://doi.org/10.48550/arXiv.1312.1685

2013-12-05

Computer Vision and Pattern Recognition

Abstract:In this paper, we present a novel Gabor wavelet based Kernel Entropy Component Analysis (KECA) method by integrating the Gabor wavelet transformation (GWT) of facial images with the KECA method for enhanced face recognition performance. Firstly, from the Gabor wavelet transformed images the most important discriminative desirable facial features characterized by spatial frequency, spatial locality and orientation selectivity to cope with the variations due to illumination and facial expression changes were derived. After that KECA, relating to the Renyi entropy is extended to include cosine kernel function. The KECA with the cosine kernels is then applied on the extracted most important discriminating feature vectors of facial images to obtain only those real kernel ECA eigenvectors that are associated with eigenvalues having positive entropy contribution. Finally, these real KECA features are used for image classification using the L1, L2 distance measures; the Mahalanobis distance measure and the cosine similarity measure. The feasibility of the Gabor based KECA method with the cosine kernel has been successfully tested on both frontal and pose-angled face recognition, using datasets from the ORL, FRAV2D and the FERET database.
A Face Recognition approach based on entropy estimate of the nonlinear DCT features in the Logarithm Domain together with Kernel Entropy Component Analysis

Arindam Kar,Debotosh Bhattacharjee,Dipak Kumar Basu,Mita Nasipuri,Mahantapas Kundu

DOI: https://doi.org/10.5815/ijitcs.2013.09.03

2013-12-05

Abstract:This paper exploits the feature extraction capabilities of the discrete cosine transform (DCT) together with an illumination normalization approach in the logarithm domain that increase its robustness to variations in facial geometry and illumination. Secondly in the same domain the entropy measures are applied on the DCT coefficients so that maximum entropy preserving pixels can be extracted as the feature vector. Thus the informative features of a face can be extracted in a low dimensional space. Finally, the kernel entropy component analysis (KECA) with an extension of arc cosine kernels is applied on the extracted DCT coefficients that contribute most to the entropy estimate to obtain only those real kernel ECA eigenvectors that are associated with eigenvalues having high positive entropy contribution. The resulting system was successfully tested on real image sequences and is robust to significant partial occlusion and illumination changes, validated with the experiments on the FERET, AR, FRAV2D and ORL face databases. Experimental comparison is demonstrated to prove the superiority of the proposed approach in respect to recognition accuracy. Using specificity and sensitivity we find that the best is achieved when Renyi entropy is applied on the DCT coefficients. Extensive experimental comparison is demonstrated to prove the superiority of the proposed approach in respect to recognition accuracy. Moreover, the proposed approach is very simple, computationally fast and can be implemented in any real-time face recognition system.

Computer Vision and Pattern Recognition
Kernel optimization-based discriminant analysis for face recognition

Jun-Bao Li,Jeng-Shyang Pan,Zhe-Ming Lu

DOI: https://doi.org/10.1007/s00521-009-0282-y

2009-01-01

Neural Computing and Applications

Abstract:The selection of kernel function and its parameter influences the performance of kernel learning machine. The difference geometry structure of the empirical feature space is achieved under the different kernel and its parameters. The traditional changing only the kernel parameters method will not change the data distribution in the empirical feature space, which is not feasible to improve the performance of kernel learning. This paper applies kernel optimization to enhance the performance of kernel discriminant analysis and proposes a so-called Kernel Optimization-based Discriminant Analysis (KODA) for face recognition. The procedure of KODA consisted of two steps: optimizing kernel and projecting. KODA automatically adjusts the parameters of kernel according to the input samples and performance on feature extraction is improved for face recognition. Simulations on Yale and ORL face databases are demonstrated the feasibility of enhancing KDA with kernel optimization.
An Efficient Reformative Kernel Discriminant Analysis for Face Recognition

Jun-Bao Li,Jeng-Shyang Pan,Zhe-Ming Lu

DOI: https://doi.org/10.1109/ROBIO.2006.340211

2006-01-01

Abstract:An efficient reformative kernel discriminant analysis, namely enhanced kernel discriminant analysis (EKDA), is proposed in this paper. In the proposed algorithm, a novel criterion, i.e., maximizing the class separability both in the feature space and in the projection subspace, is presented to enhance the discriminant power of KDA. EKDA is more adaptive to the input data under the novel criterion compared with KDA, which enhances the performance of EKDA. Experiments conducted on the Yale and ORL face databases give the higher recognition performance compared with KDA.
Kernel Least-Mean Mixed-Norm Algorithm

Miao, Q.Y.,Li, C.G.

DOI: https://doi.org/10.1049/cp.2012.1214

2012-01-01

Abstract:The Kernel method is a powerful tool for extending an algorithm from linear to nonlinear case. The least-mean mixed-norm (LMMN) algorithm possesses good performance when the system measurement noise shows distribution with a linear combination of long tails and short tails. In this paper, we combine the famed kernel trick and the LMMN algorithm to present the kernel LMMN (KLMMN) algorithm, which is an adaptive filtering algorithm in reproducing kernel Hilbert space (RKHS). The optimal norm-mixing parameter is derived. To demonstrate the effectiveness and superiorities of the proposed algorithm, we apply the algorithm to nonlinear system identification when the environment noise composed of a linear combination of Gaussian and Bernoulli distributions.
A Novel Kernel-Based Maximum a Posteriori Classification Method

Zenglin Xu,Kaizhu Huang,Jianke Zhu,Irwin King,Michael R Lyu

DOI: https://doi.org/10.1016/j.neunet.2008.11.005

2008-01-01

Abstract:Kernel methods have been widely used in pattern recognition. Many kernel classifiers such as Support Vector Machines (SVM) assume that data can be separated by a hyperplane in the kernel-induced feature space. These methods do not consider the data distribution and are difficult to output the probabilities or confidences for classification. This paper proposes a novel Kernel-based Maximum A Posteriori (KMAP) classification method, which makes a Gaussian distribution assumption instead of a linear separable assumption in the feature space. Robust methods are further proposed to estimate the probability densities, and the kernel trick is utilized to calculate our model. The model is theoretically and empirically important in the sense that: (1) it presents a more generalized classification model than other kernel-based algorithms, e.g., Kernel Fisher Discriminant Analysis (KFDA); (2) it can output probability or confidence for classification, therefore providing potential for reasoning under uncertainty; and (3) multi-way classification is as straightforward as binary classification in this model, because only probability calculation is involved and no one-against-one or one-against-others voting is needed. Moreover, we conduct an extensive experimental comparison with state-of-the-art classification methods, such as SVM and KFDA, on both eight UCI benchmark data sets and three face data sets. The results demonstrate that KMAP achieves very promising performance against other models.
Face recognition using a kernel fractional-step discriminant analysis algorithm

Guang Dai,Dit-Yan Yeung,Yun-Tao Qian

DOI: https://doi.org/10.1016/j.patcog.2006.06.030

IF: 8

2007-01-01

Pattern Recognition

Abstract:Feature extraction is among the most important problems in face recognition systems. In this paper, we propose an enhanced kernel discriminant analysis (KDA) algorithm called kernel fractional-step discriminant analysis (KFDA) for nonlinear feature extraction and dimensionality reduction. Not only can this new algorithm, like other kernel methods, deal with nonlinearity required for many face recognition tasks, it can also outperform traditional KDA algorithms in resisting the adverse effects due to outlier classes. Moreover, to further strengthen the overall performance of KDA algorithms for face recognition, we propose two new kernel functions: cosine fractional-power polynomial kernel and non-normal Gaussian RBF kernel. We perform extensive comparative studies based on the YaleB and FERET face databases. Experimental results show that our KFDA algorithm outperforms traditional kernel principal component analysis (KPCA) and KDA algorithms. Moreover, further improvement can be obtained when the two new kernel functions are used.
Classification algorithm of hyperspectral images based on kernel entropy analysis

王瀛,郭雷,梁楠

DOI: https://doi.org/10.13229/j.cnki.jdxbgxb2012.06.033

2012-01-01

Abstract:To take advantage of the characteristics of KECA for hyperspectral remote sensing image classification,an approach of sample set selection and C-means classification is proposed.The sample selection is based on convex geometry concepts and C-means classification uses spectral angles as distance metrics in feature space.Experiment results of HYDICE hyperspectral data confirm that the proposed approach can improve classification accuracy effectively.
Kernel Generalized Nonlinear Discriminant Analysis Algorithm For Pattern Recognition

Guang Dai,Yuntao Qian

DOI: https://doi.org/10.1109/ICIP.2004.1421660

2004-01-01

Abstract:Linear discriminant analysis (LDA) is a very effective tool used for dimensionality reduction and feature extraction in pattern recognition. However, the LDA is inadequate to describe complex and nonlinear patterns. To solve this problem, kernel nonlinear discriminant analysis (K-NDA) has been proposed. Although successful in many cases, classic K-NDA also suffers from the small sample size problem (SSSP) and loses some significant discriminatory information as same as classic LDA. In this paper. a novel K-NDA, i.e., the kernel generalized nonlinear discriminant analysis (KG-NDA) algorithm is introduced to effectively overcome these problems, and it also views the optimal discriminant vectors as a global transform in the feature space to some extent. It not only deals with the nonlinear problem, but also effectively solves the SSSP. The KG-NDA is applied to the experiments oil face recognition, and the results tested oil two popular databases demonstrate that this method is very effective.
Adaptive quasiconformal kernel discriminant analysis

Pan Jeng-Shyang,Li Jun-Bao,Lu Zhe-Ming

DOI: https://doi.org/10.1016/j.neucom.2007.10.003

IF: 6

2008-01-01

Neurocomputing

Abstract:Kernel discriminant analysis (KDA) is effective to extract nonlinear discriminative features of input samples using the kernel trick. However, the conventional KDA algorithm endures the kernel selection which has significant impact on the performances of KDA. In order to overcome this limitation, a novel nonlinear feature extraction method called adaptive quasiconformal kernel discriminant analysis (AQKDA) is proposed in this paper. AQKDA maps the data from the original input space to the high dimensional kernel space using a quasiconformal kernel. The adaptive parameters of the quasiconformal kernel are automatically calculated through optimizing an objective function designed for measuring the class separability of data in the feature space. Consequently, the nonlinear features extracted by AQKDA have the larger class separability compared with KDA. Experimental results on the two real-world datasets demonstrate the effectiveness of the proposed method. © 2007 Elsevier B.V. All rights reserved.
Coupled kernel embedding for low resolution face image recognition

Chuan-Xian Ren,Dao-Qing Dai,Hong Yan

DOI: https://doi.org/10.1109/TIP.2012.2192285

Abstract:Practical video scene and face recognition systems are sometimes confronted with low-resolution (LR) images. The faces may be very small even if the video is clear, thus it is difficult to directly measure the similarity between the faces and the high-resolution (HR) training samples. Traditional super-resolution (SR) methods based face recognition usually have limited performance because the target of SR may not be consistent with that of classification, and time-consuming SR algorithms are not suitable for real-time applications. In this paper, a new feature extraction method called Coupled Kernel Embedding (CKE) is proposed for LR face recognition without any SR preprocessing. In this method, the final kernel matrix is constructed by concatenating two individual kernel matrices in the diagonal direction, and the (semi-)positively definite properties are preserved for optimization. CKE addresses the problem of comparing multi-modal data that are difficult for conventional methods in practice due to the lack of an efficient similarity measure. Particularly, different kernel types (e.g., linear, Gaussian, polynomial) can be integrated into an uniformed optimization objective, which cannot be achieved by simple linear methods. CKE solves this problem by minimizing the dissimilarities captured by their kernel Gram matrices in the low- and high-resolution spaces. In the implementation, the nonlinear objective function is minimized by a generalized eigenvalue decomposition. Experiments on benchmark and real databases show that our CKE method indeed improves the recognition performance.
Sparse Kernel Canonical Correlation Analysis Via $\Ell_1$-Regularization

Xiaowei Zhang,Delin Chu,Li-Zhi Liao,Michael K. Ng

DOI: https://doi.org/10.48550/arxiv.1701.04207

2017-01-01

Abstract:Canonical correlation analysis (CCA) is a multivariate statistical technique for finding the linear relationship between two sets of variables. The kernel generalization of CCA named kernel CCA has been proposed to find nonlinear relations between datasets. Despite their wide usage, they have one common limitation that is the lack of sparsity in their solution. In this paper, we consider sparse kernel CCA and propose a novel sparse kernel CCA algorithm (SKCCA). Our algorithm is based on a relationship between kernel CCA and least squares. Sparsity of the dual transformations is introduced by penalizing the $\ell_{1}$-norm of dual vectors. Experiments demonstrate that our algorithm not only performs well in computing sparse dual transformations but also can alleviate the over-fitting problem of kernel CCA.
Optimal Discriminant Analysis Based on Kernel Extension of Graph Embedding and Face Recognition

LU Gui-Fu,LIN Zhong,JIN Zhong

DOI: https://doi.org/10.3724/sp.j.1001.2011.03843

2011-01-01

Journal of Software

Abstract:By making use of compressive mapping and isomorphic mapping in the kernel extension of graph embedding, this paper proves that the essence of kernel extension of graph embedding (KGE) is KPCA (kernel principal component analysis) plus all kinds of linear dimension reduction approaches interpreted in a linear extension of graph embedding (LGE).Based on the theory framework, a combined framework, which takes advantage of the discriminant feature in both null and non-null spaces, is developed.Furthermore, every kernel dimensionality reduction algorithm has its own corresponding combined algorithm.The experimental results from ORL, Yale, FERET and PIE face databases show that the proposed methods are better than the original methods in terms of recognition rate.
Kernel linear regression for face recognition

Yuwu Lu,Xiaozhao Fang,Binglei Xie

DOI: https://doi.org/10.1007/s00521-013-1435-6

2013-06-14

Neural Computing and Applications

Abstract:Linear regression uses the least square algorithm to solve the solution of linear regression equation. Linear regression classification (LRC) shows good classification performance on face image data. However, when the axes of linear regression of class-specific samples have intersections, LRC could not well classify the samples that distribute around intersections. Moreover, the LRC could not perform well at the situation of severe lighting variations. This paper proposes a new classification method, kernel linear regression classification (KLRC), based on LRC and the kernel trick. KLRC is a nonlinear extension of LRC and can offset the drawback of LRC. KLRC implicitly maps the data into a high-dimensional kernel space by using the nonlinear mapping determined by a kernel function. Through this mapping, KLRC is able to make the data more linearly separable and can perform well for face recognition with varying lighting. For comparison, we conduct on three standard databases under some evaluation protocols. The proposed methodology not only outperforms LRC but also takes the better performance than typical kernel methods such as kernel linear discriminant analysis and kernel principal component analysis.

computer science, artificial intelligence
Fast and Scalable Multi-Kernel Encoder Classifier

Cencheng Shen

2024-06-04

Abstract:This paper introduces a new kernel-based classifier by viewing kernel matrices as generalized graphs and leveraging recent progress in graph embedding techniques. The proposed method facilitates fast and scalable kernel matrix embedding, and seamlessly integrates multiple kernels to enhance the learning process. Our theoretical analysis offers a population-level characterization of this approach using random variables. Empirically, our method demonstrates superior running time compared to standard approaches such as support vector machines and two-layer neural network, while achieving comparable classification accuracy across various simulated and real datasets.

Machine Learning
EMPIRICAL LIKELIHOOD‐BASED KERNEL DENSITY ESTIMATION

SX Chen

DOI: https://doi.org/10.1111/j.1467-842x.1997.tb00522.x

1997-01-01

Australian Journal of Statistics

Abstract:summaryThis paper considers the estimation of a probability density function when extra distributional information is available (e.g. the mean of the distribution is known or the variance is a known function of the mean). The standard kernel method cannot exploit such extra information systematically as it uses an equal probability weight n‐1 at each data point. The paper suggests using empirical likelihood to choose the probability weights under constraints formulated from the extra distributional information. An empirical likelihood‐based kernel density estimator is given by replacing n‐1 by the empirical likelihood weights, and has these advantages: it makes systematic use of the extra information, it is able to reflect the extra characteristics of the density function, and its variance is smaller than that of the standard kernel density estimator.

L1-norm-based Kernel Entropy Components

Optimized Kernel Entropy Components

Improved KECA for Process Monitoring Using Ensemble Learning and Bayesian Inference

Sparse Kernel Entropy Component Analysis for Dimensionality Reduction of Biomedical Data

L1-Norm Kernel Discriminant Analysis Via Bayes Error Bound Optimization for Robust Feature Extraction

Human Face Recognition using Gabor based Kernel Entropy Component Analysis

A Face Recognition approach based on entropy estimate of the nonlinear DCT features in the Logarithm Domain together with Kernel Entropy Component Analysis

Kernel optimization-based discriminant analysis for face recognition

An Efficient Reformative Kernel Discriminant Analysis for Face Recognition

Kernel Least-Mean Mixed-Norm Algorithm

A Novel Kernel-Based Maximum a Posteriori Classification Method

Face recognition using a kernel fractional-step discriminant analysis algorithm

Classification algorithm of hyperspectral images based on kernel entropy analysis

Kernel Generalized Nonlinear Discriminant Analysis Algorithm For Pattern Recognition

Adaptive quasiconformal kernel discriminant analysis

Coupled kernel embedding for low resolution face image recognition

Sparse Kernel Canonical Correlation Analysis Via $\Ell_1$-Regularization

Optimal Discriminant Analysis Based on Kernel Extension of Graph Embedding and Face Recognition

Kernel linear regression for face recognition

Fast and Scalable Multi-Kernel Encoder Classifier

EMPIRICAL LIKELIHOOD‐BASED KERNEL DENSITY ESTIMATION