A Majorization-Minimization Approach to Lq Norm Multiple Kernel Learning

Zhizheng Liang,Shixiong Xia,Jin Liu,Yong Zhou,Lei Zhang

DOI: https://doi.org/10.1109/ACPR.2013.54

IF: 8

2013-01-01

Pattern Recognition

Abstract:Multiple kernel learning (MKL) usually searches for linear (nonlinear) combinations of predefined kernels by optimizing some performance measures. However, previous MKL algorithms cannot deal with Lq norm MKL if q

What problem does this paper attempt to address?

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Lin Luo,Hongye Su,Baofen Zheng,Junfeng Zhang

DOI: https://doi.org/10.3233/ifs-141149

2014-01-01

Journal of Intelligent & Fuzzy Systems

Abstract:In many applications of kernel method, such as support vector machine (SVM), the performance greatly depends on the choice of kernel. It is not always clear what is the most suitable kernel for one specified task. Multiple kernel learning (MKL) allows to optimize over linear combinations of kernels. A two-step approach, which alternately optimizes the standard SVM and kernel weights, is proposed to solve the non-convex MKL problem. The generalized convexity of MKL is studied which guarantees the strong duality of the corresponding optimization problem. A further contribution is the utilization of a computationally efficient inexact-projection-based method to optimize the standard SVM. In addition, a nonmonotone gradient method is proposed to optimize the kernel weights and the Hessian matrix is approximated with a variation of Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) approach. The global convergence of the proposed two-step approach is analyzed. The utility of the proposed scheme is demonstrated by empirical study with several datasets, and its performance is compared with state-of-the-art methods in terms of accuracy and scalability. The results show that the proposed method performs as well as existing methods in accuracy, but takes much less time to converge to the stationary point.
ℓq-Norm Sample-Adaptive Multiple Kernel Learning

Qiang Wang,Xinwang Liu,Jiaqing Xu

DOI: https://doi.org/10.1109/access.2020.2974903

IF: 3.9

2020-01-01

IEEE Access

Abstract:Existing multiple kernel learning (MKL) algorithms indiscriminately apply the same set of kernel combination weights to all samples by pre-specifying a group of base kernels. Sample-adaptive MKL learning (SAMKL) overcomes this limitation by adaptively switching on/off the base kernels with respect to each sample. However, it restricts to solving MKL problems with pre-specified kernels. And, the formulation of existing SAMKL falls to an ℓ 1 -norm MKL which is not flexible. To allow for robust kernel mixtures that generalize well in practical applications, we extend SAMKL to the arbitrary norm and apply it to image classification. In this paper, we formulate a closed-form solution for optimizing the kernel weights based on the equivalence between group-lasso and MKL, and derive an efficient ℓ q -norm (q ≥ 1 and denoting the ℓ q -norm of kernel weights) SAMKL algorithm. The cutting plane method is used to solve this margin maximization problem. Besides, we propose a framework for solving MKL problems in image classification. Experimental results on multiple data sets show the promising performance of the proposed solution compared with other competitive methods.
A Multiple Kernel Learning Model Based on P-Norm

Jinshan Qi,Xun Liang,Rui Xu

DOI: https://doi.org/10.1155/2018/1018789

IF: 3.12

2018-01-01

Computational Intelligence and Neuroscience

Abstract:By utilizing kernel functions, support vector machines (SVMs) successfully solve the linearly inseparable problems. Subsequently, its applicable areas have been greatly extended. Using multiple kernels (MKs) to improve the SVM classification accuracy has been a hot topic in the SVM research society for several years. However, most MK learning (MKL) methods employ L1-norm constraint on the kernel combination weights, which forms a sparse yet nonsmooth solution for the kernel weights. Alternatively, the Lp-norm constraint on the kernel weights keeps all information in the base kernels. Nonetheless, the solution of Lp-norm constraint MKL is nonsparse and sensitive to the noise. Recently, some scholars presented an efficient sparse generalized MKL (L1- and L2-norms based GMKL) method, in which L1 L2 established an elastic constraint on the kernel weights. In this paper, we further extend the GMKL to a more generalized MKL method based on the p-norm, by joining L1- and Lp-norms. Consequently, the L1- and L2-norms based GMKL is a special case in our method when p=2. Experiments demonstrated that our L1- and Lp-norms based MKL offers a higher accuracy than the L1- and L2-norms based GMKL in the classification, while keeping the properties of the L1- and L2-norms based on GMKL.
Efficient sparse generalized multiple kernel learning

Haiqin Yang,Zenglin Xu,Jieping Ye,Irwin King,Michael R Lyu

DOI: https://doi.org/10.1109/TNN.2010.2103571

Abstract:Kernel methods have been successfully applied in various applications. To succeed in these applications, it is crucial to learn a good kernel representation, whose objective is to reveal the data similarity precisely. In this paper, we address the problem of multiple kernel learning (MKL), searching for the optimal kernel combination weights through maximizing a generalized performance measure. Most MKL methods employ the L(1)-norm simplex constraints on the kernel combination weights, which therefore involve a sparse but non-smooth solution for the kernel weights. Despite the success of their efficiency, they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance. Alternatively, imposing the L(p)-norm (p > 1) constraint on the kernel weights will keep all the information in the base kernels. This leads to non-sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information. To tackle these problems, we propose a generalized MKL (GMKL) model by introducing an elastic-net-type constraint on the kernel weights. More specifically, it is an MKL model with a constraint on a linear combination of the L(1)-norm and the squared L(2)-norm on the kernel weights to seek the optimal kernel combination weights. Therefore, previous MKL problems based on the L(1)-norm or the L(2)-norm constraints can be regarded as special cases. Furthermore, our GMKL enjoys the favorable sparsity property on the solution and also facilitates the grouping effect. Moreover, the optimization of our GMKL is a convex optimization problem, where a local solution is the global optimal solution. We further derive a level method to efficiently solve the optimization problem. A series of experiments on both synthetic and real-world datasets have been conducted to show the effectiveness and efficiency of our GMKL.
Training Lp Norm Multiple Kernel Learning in the Primal

Zhizheng Liang,Shixiong Xia,Yong Zhou,Lei Zhang

DOI: https://doi.org/10.1016/j.neunet.2013.05.003

IF: 7.8

2013-01-01

Neural Networks

Abstract:Some multiple kernel learning (MKL) models are usually solved by utilizing the alternating optimization method where one alternately solves SVMs in the dual and updates kernel weights. Since the dual and primal optimization can achieve the same aim, it is valuable in exploring how to perform Lp norm MKL in the primal. In this paper, we propose an Lp norm multiple kernel learning algorithm in the primal where we resort to the alternating optimization method: one cycle for solving SVMs in the primal by using the preconditioned conjugate gradient method and other cycle for learning the kernel weights. It is interesting to note that the kernel weights in our method can obtain analytical solutions. Most importantly, the proposed method is well suited for the manifold regularization framework in the primal since solving LapSVMs in the primal is much more effective than solving LapSVMs in the dual. In addition, we also carry out theoretical analysis for multiple kernel learning in the primal in terms of the empirical Rademacher complexity. It is found that optimizing the empirical Rademacher complexity may obtain a type of kernel weights. The experiments on some datasets are carried out to demonstrate the feasibility and effectiveness of the proposed method.
Efficient Multiple Kernel Learning Algorithms Using Low-Rank Representation.

Wenjia Niu,Kewen Xia,Baokai Zu,Jianchuan Bai

DOI: https://doi.org/10.1155/2017/3678487

IF: 3.12

2017-01-01

Computational Intelligence and Neuroscience

Abstract:Unlike Support Vector Machine (SVM), Multiple Kernel Learning (MKL) allows datasets to be free to choose the useful kernels based on their distribution characteristics rather than a precise one. It has been shown in the literature that MKL holds superior recognition accuracy compared with SVM, however, at the expense of time consuming computations. This creates analytical and computational difficulties in solving MKL algorithms. To overcome this issue, we first develop a novel kernel approximation approach for MKL and then propose an efficient Low-Rank MKL (LR-MKL) algorithm by using the Low-Rank Representation (LRR). It is well-acknowledged that LRR can reduce dimension while retaining the data features under a global low-rank constraint. Furthermore, we redesign the binary-class MKL as the multiclass MKL based on pairwise strategy. Finally, the recognition effect and efficiency of LR-MKL are verified on the datasets Yale, ORL, LSVT, and Digit. Experimental results show that the proposed LR-MKL algorithm is an efficient kernel weights allocation method in MKL and boosts the performance of MKL largely.
Selective multiple kernel learning for classification with ensemble strategy.

Tao Sun,Licheng Jiao,Fang Liu,Shuang Wang,Jie Feng

DOI: https://doi.org/10.1016/j.patcog.2013.04.003

IF: 8

2013-01-01

Pattern Recognition

Abstract:Multiple Kernel Learning (MKL) aims to seek a better result than single kernel learning by combining a compact set of sub-kernels. However, MKL with L1-norm easily discards the sub-kernels with complementary information and MKL with Lp-norm(p≥2) often gets the redundant solution. To address these problems, a Selective Multiple Kernel Learning (SMKL) method, inspired by Ensemble Learning (EL), is proposed. Comparing MKL with Lp-norm(p≥2), SMKL obtains a sparse solution by a pre-selection procedure. Comparing MKL with L1-norm, SMKL preserves the sub-kernels with complementary information by guaranteeing the high discrimination and large diversity of pre-selected sub-kernels. For quantifying the discrimination and diversity of sub-kernels, a new kernel evaluation is designed. SMKL reduces the scale of MKL optimization and saves the memory storing of the sub-kernels, which extends the scale of problem that MKL could solve. Specially, a fast SMKL method using L∞-norm constraint is focused, which needs no MKL optimization process. It means that the memory is hardly a limitation for MKL with the large scale problem. Experiments state that our method is effective for classification.
Matrix-Regularized Multiple Kernel Learning Via &Lt;inline-Formula> &Lt;tex-Math Notation="latex">$(r,~p)$ &Lt;/tex-Math> &Lt;/inline-Formula> Norms

Yina Han,Yixin Yang,Xuelong Li,Qingyu Liu,Yuanliang Ma

DOI: https://doi.org/10.1109/tnnls.2017.2785329

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper examines a matrix-regularized multiple kernel learning (MKL) technique based on a notion of (r, p) norms. For the problem of learning a linear combination in the support vector machine-based framework, model complexity is typically controlled using various regularization strategies on the combined kernel weights. Recent research has developed a generalized ℓ p -norm MKL framework with tunable variable p(p ≥ 1) to support controlled intrinsic sparsity. Unfortunately, this “1-D” vector ℓ p -norm hardly exploits potentially useful information on how the base kernels “interact.” To allow for higher order kernel-pair relationships, we extend the “1-D” vector ℓ p -MKL to the “2-D” matrix (r, p) norms (1 ≤ r, p <; ∞). We develop a new formulation and an efficient optimization strategy for (r, p)-MKL with guaranteed convergence. A theoretical analysis and experiments on seven UCI data sets shed light on the superiority of (r, p)-MKL over ℓ p -MKL in various scenarios.
Unsupervised Multiple Kernel Learning

Jialei Wang,Jinfeng Zhuang,Steven C. H. Hoi

2011-01-01

Abstract:Traditional multiple kernel learning (MKL) algorithms are essentially supervised learning in the sense that the kernel learning task requires the class labels of training data. However, class labels may not always be available prior to the kernel learning task in some real world scenarios, e.g., an early preprocessing step of a classification task or an unsupervised learning task such as dimension reduction. In this paper, we investigate a problem of Unsupervised Multiple Kernel Learning (UMKL), which does not require class labels of training data as needed in a conventional multiple kernel learning task. Since a kernel essentially defines pairwise similarity between any two examples, our unsupervised kernel learning method mainly follows two intuitive principles: (1) a good kernel should allow every example to be well reconstructed from its localized bases weighted by the kernel values; (2) a good kernel should induce kernel values that are coincided with the local geometry of the data. We formulate the unsupervised multiple kernel learning problem as an optimization task and propose an efficient alternating optimization algorithm to solve it. Empirical results on both classification and dimension reductions tasks validate the efficacy of the proposed UMKL algorithm.
Multiple Kernel Learning Method Using MRMR Criterion and Kernel Alignment

Peng Wu,Fuqing Duan,Ping Guo

DOI: https://doi.org/10.1007/978-3-642-42054-2_15

2013-01-01

Abstract:Multiple kernel learning (MKL) is a widely used kernel learning method, but how to select kernel is lack of theoretical guidance. The performance of MKL is depend on the users’ experience, which is difficult to choose the proper kernels in practical applications. In this paper, we propose a MKL method based on minimal redundant maximal relevance criterion and kernel alignment. The main feature of this method compared to others in the literature is that the selection of kernels is considered as a feature selection issue in the Hilbert space, and can obtain a set of base kernels with the highest relevance to the target task and the minimal redundancies among themselves. Experimental results on several benchmark classification data sets show that our proposed method can enhance the performance of MKL.
Kernel Least-Mean Mixed-Norm Algorithm

Miao, Q.Y.,Li, C.G.

DOI: https://doi.org/10.1049/cp.2012.1214

2012-01-01

Abstract:The Kernel method is a powerful tool for extending an algorithm from linear to nonlinear case. The least-mean mixed-norm (LMMN) algorithm possesses good performance when the system measurement noise shows distribution with a linear combination of long tails and short tails. In this paper, we combine the famed kernel trick and the LMMN algorithm to present the kernel LMMN (KLMMN) algorithm, which is an adaptive filtering algorithm in reproducing kernel Hilbert space (RKHS). The optimal norm-mixing parameter is derived. To demonstrate the effectiveness and superiorities of the proposed algorithm, we apply the algorithm to nonlinear system identification when the environment noise composed of a linear combination of Gaussian and Bernoulli distributions.
A Primal Method for Multiple Kernel Learning

Zhifeng Hao,Ganzhao Yuan,Xiaowei Yang,Zijie Chen

DOI: https://doi.org/10.1007/s00521-012-1022-2

2012-01-01

Abstract:The canonical support vector machines (SVMs) are based on a single kernel, recent publications have shown that using multiple kernels instead of a single one can enhance interpretability of the decision function and promote classification accuracy. However, most of existing approaches mainly reformulate the multiple kernel learning as a saddle point optimization problem which concentrates on solving the dual. In this paper, we show that the multiple kernel learning (MKL) problem can be reformulated as a BiConvex optimization and can also be solved in the primal. While the saddle point method still lacks convergence results, our proposed method exhibits strong optimization convergence properties. To solve the MKL problem, a two-stage algorithm that optimizes canonical SVMs and kernel weights alternately is proposed. Since standard Newton and gradient methods are too time-consuming, we employ the truncated-Newton method to optimize the canonical SVMs. The Hessian matrix need not be stored explicitly, and the Newton direction can be computed using several Preconditioned Conjugate Gradient steps on the Hessian operator equation, the algorithm is shown more efficient than the current primal approaches in this MKL setting. Furthermore, we use the Nesterov’s optimal gradient method to optimize the kernel weights. One remarkable advantage of solving in the primal is that it achieves much faster convergence rate than solving in the dual and does not require a two-stage algorithm even for the single kernel LapSVM. Introducing the Laplacian regularizer, we also extend our primal method to semi-supervised scenario. Extensive experiments on some UCI benchmarks have shown that the proposed algorithm converges rapidly and achieves competitive accuracy.
Absent Multiple Kernel Learning Algorithms.

Xinwang Liu,Lei Wang,Xinzhong Zhu,Miaomiao Li,En Zhu,Tongliang Liu,Li Liu,Yong Dou,Jianping Yin

DOI: https://doi.org/10.1109/tpami.2019.2895608

IF: 23.6

2019-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Multiple kernel learning (MKL) has been intensively studied during the past decade. It optimally combines the multiple channels of each sample to improve classification performance. However, existing MKL algorithms cannot effectively handle the situation where some channels of the samples are missing, which is not uncommon in practical applications. This paper proposes three absent MKL (AMKL) algorithms to address this issue. Different from existing approaches where missing channels are first imputed and then a standard MKL algorithm is deployed on the imputed data, our algorithms directly classify each sample based on its observed channels, without performing imputation. Specifically, we define a margin for each sample in its own relevant space, a space corresponding to the observed channels of that sample. The proposed AMKL algorithms then maximize the minimum of all sample-based margins, and this leads to a difficult optimization problem. We first provide two two-step iterative algorithms to approximately solve this problem. After that, we show that this problem can be reformulated as a convex one by applying the representer theorem. This makes it readily be solved via existing convex optimization packages. In addition, we provide a generalization error bound to justify the proposed AMKL algorithms from a theoretical perspective. Extensive experiments are conducted on nine UCI and six MKL benchmark datasets to compare the proposed algorithms with existing imputation-based methods. As demonstrated, our algorithms achieve superior performance and the improvement is more significant with the increase of missing ratio.
Multiple Empirical Kernel Learning with Locality Preserving Constraint

Qi Fan,Daqi Gao,Zhe Wang

DOI: https://doi.org/10.1016/j.knosys.2016.05.008

IF: 8.139

2016-01-01

Knowledge-Based Systems

Abstract:Multiple Kernel Learning (MKL) is flexible in dealing with problems involving multiple and heterogeneous data sources. However, the necessity of inner-product form restricts its application since to kernelize the algorithms unsatisfying the inner-product formulation is pretty difficult. To overcome this problem, Multiple Empirical Kernel Learning (MEKL) is proposed by explicitly mapping input samples to feature spaces, in which the mapped feature vectors are explicitly presented. Most existed MEKLs optimize the learning framework by minimizing empirical risk, regularization risk and the loss term of multiple feature spaces. As little attention is paid to preserving local structure among training samples, the learned classifier might lack of locality similarity preserving property, which might result in unfavorable performance. Inspired by Locality Preserving Projection (LPP) which is to seek the optimal projection by preserving the local property of input samples, we introduce the locality preserving constraint into the learning framework to propose a novel Multiple Empirical Kernel Learning with Locality Preserving Constraint (MEKL-LPC). MEKL-LPC shows lower generalization error bound than both the Modification of Ho–Kashyap algorithm with Squared approximation of the misclassification error (MHKS) and Multi-Kernel MHKS (MultiK-MHKS) in terms of Rademacher complexity. Experiments on several real-world datasets demonstrate that MEKL-LPC outperforms the compared algorithms. The contributions of this work are: (i) originally integrating locality preserving constraint into MEKL, (ii) proposing a lower generalization error bound algorithm, i.e. MEKL-LPC.
Joint Learning of Distance Metric and Kernel Classifier via Multiple Kernel Learning.

Weiqi Zhang,Zifei Yan,Hongzhi Zhang,Wangmeng Zuo

DOI: https://doi.org/10.1007/978-981-10-3002-4_48

2016-01-01

Abstract:Both multiple kernel learning (MKL) and support vector metric learning (SVML) were developed to adaptively learn kernel function from training data, and have been proved to be effective in many challenging applications. Actually, many MKL formulations are based on either the max-margin or radius-margin based principles, which in spirit is consistent with the optimization principle of the between/within class distances adopted in SVML. This motivates us to investigate their connection and develop a novel model for joint learning distance metric and kernel classifier. In this paper, we provide a new parameterization scheme for incorporating the squared Mahalanobis distance into the Gaussian RBF kernel, and formulate kernel learning into a GMKL framework. Moreover, radius information is also incorporated as the supplement for considering the within-class distance in the feature space. We demonstrate the effectiveness of the proposed algorithm on several benchmark datasets of varying sizes and difficulties. Experimental results show that the proposed algorithm achieves competitive classification accuracies with both state-of-the-art metric learning models and representative kernel learning models.
Multiple Kernel Subspace Clustering with Local Structural Graph and Low-Rank Consensus Kernel Learning

Zhenwen Ren,Haoran Li,Chao Yang,Quansen Sun

DOI: https://doi.org/10.1016/j.knosys.2019.105040

IF: 8.139

2020-01-01

Knowledge-Based Systems

Abstract:Multiple kernel learning (MKL) methods are generally believed to perform better than single kernel learning (SKL) methods in handling nonlinear subspace clustering problem, largely thanks to MKL avoids selecting and tuning a pre-defined kernel. However, previous MKL methods mainly focused on how to define a kernel weighting strategy, but ignored the structural characteristics of the input data in both the original space and the kernel space. In this paper, we first propose a novel graph-based MKL method for subspace clustering, namely, Local Structural Graph and Low-Rank Consensus Multiple Kernel Learning (LLMKL). It jointly learns an optimal affinity graph and a suitable consensus kernel for clustering purpose by elegantly integrating the MKL technology, the global structure in the kernel space, the local structure in the original space, and the Hilbert space self-expressiveness property in a unified optimization model. In particular, to capture the data global structure, we employ a substitute of the desired consensus kernel, and then introduce a low-rank constraint on the substitute to encourage that the structure of linear subspaces is present in the feature space. Moreover, the data local structure is explored by building a complete graph, where each sample is treated as a node, and an edge codes the pairwise affinity between two samples. By such, the consensus kernel learning and the affinity graph learning can promote each other such that the data in resulting Hilbert space are both self-expressive and low-rank. Experiments on both image and text clustering well demonstrate that LLMKL outperforms the state-of-the-art methods.
Reduced Multiple Empirical Kernel Learning Machine

Zhe Wang,MingZhe Lu,Daqi Gao

DOI: https://doi.org/10.1007/s11571-014-9304-2

2015-01-01

Cognitive Neurodynamics

Abstract:Multiple kernel learning (MKL) is demonstrated to be flexible and effective in depicting heterogeneous data sources since MKL can introduce multiple kernels rather than a single fixed kernel into applications. However, MKL would get a high time and space complexity in contrast to single kernel learning, which is not expected in real-world applications. Meanwhile, it is known that the kernel mapping ways of MKL generally have two forms including implicit kernel mapping and empirical kernel mapping (EKM), where the latter is less attracted. In this paper, we focus on the MKL with the EKM, and propose a reduced multiple empirical kernel learning machine named RMEKLM for short. To the best of our knowledge, it is the first to reduce both time and space complexity of the MKL with EKM. Different from the existing MKL, the proposed RMEKLM adopts the Gauss Elimination technique to extract a set of feature vectors, which is validated that doing so does not lose much information of the original feature space. Then RMEKLM adopts the extracted feature vectors to span a reduced orthonormal subspace of the feature space, which is visualized in terms of the geometry structure. It can be demonstrated that the spanned subspace is isomorphic to the original feature space, which means that the dot product of two vectors in the original feature space is equal to that of the two corresponding vectors in the generated orthonormal subspace. More importantly, the proposed RMEKLM brings a simpler computation and meanwhile needs a less storage space, especially in the processing of testing. Finally, the experimental results show that RMEKLM owns a much efficient and effective performance in terms of both complexity and classification. The contributions of this paper can be given as follows: (1) by mapping the input space into an orthonormal subspace, the geometry of the generated subspace is visualized; (2) this paper first reduces both the time and space complexity of the EKM-based MKL; (3) this paper adopts the Gauss Elimination, one of the on-the-shelf techniques, to generate a basis of the original feature space, which is stable and efficient.
L2-norm multiple kernel learning and its application to biomedical data fusion

Shi Yu,Tillmann Falck,Anneleen Daemen,Leon-Charles Tranchevent,Johan Ak Suykens,Bart De Moor,Yves Moreau

DOI: https://doi.org/10.1186/1471-2105-11-309

IF: 3.307

2010-06-08

BMC Bioinformatics

Abstract:Background: This paper introduces the notion of optimizing different norms in the dual problem of support vector machines with multiple kernels. The selection of norms yields different extensions of multiple kernel learning (MKL) such as L(infinity), L1, and L2 MKL. In particular, L2 MKL is a novel method that leads to non-sparse optimal kernel coefficients, which is different from the sparse kernel coefficients optimized by the existing L(infinity) MKL method. In real biomedical applications, L2 MKL may have more advantages over sparse integration method for thoroughly combining complementary information in heterogeneous data sources. Results: We provide a theoretical analysis of the relationship between the L2 optimization of kernels in the dual problem with the L2 coefficient regularization in the primal problem. Understanding the dual L2 problem grants a unified view on MKL and enables us to extend the L2 method to a wide range of machine learning problems. We implement L2 MKL for ranking and classification problems and compare its performance with the sparse L(infinity) and the averaging L1 MKL methods. The experiments are carried out on six real biomedical data sets and two large scale UCI data sets. L2 MKL yields better performance on most of the benchmark data sets. In particular, we propose a novel L2 MKL least squares support vector machine (LSSVM) algorithm, which is shown to be an efficient and promising classifier for large scale data sets processing. Conclusions: This paper extends the statistical framework of genomic data fusion based on MKL. Allowing non-sparse weights on the data sources is an attractive option in settings where we believe most data sources to be relevant to the problem at hand and want to avoid a "winner-takes-all" effect seen in L(infinity) MKL, which can be detrimental to the performance in prospective studies. The notion of optimizing L2 kernels can be straightforwardly extended to ranking, classification, regression, and clustering algorithms. To tackle the computational burden of MKL, this paper proposes several novel LSSVM based MKL algorithms. Systematic comparison on real data sets shows that LSSVM MKL has comparable performance as the conventional SVM MKL algorithms. Moreover, large scale numerical experiments indicate that when cast as semi-infinite programming, LSSVM MKL can be solved more efficiently than SVM MKL. Availability: The MATLAB code of algorithms implemented in this paper is downloadable from http://homes.esat.kuleuven.be/~sistawww/bioi/syu/l2lssvm.html.
A Unified View of Localized Kernel Learning

John Moeller,Sarathkrishna Swaminathan,Suresh Venkatasubramanian

DOI: https://doi.org/10.48550/arXiv.1603.01374

2016-03-04

Abstract:Multiple Kernel Learning, or MKL, extends (kernelized) SVM by attempting to learn not only a classifier/regressor but also the best kernel for the training task, usually from a combination of existing kernel functions. Most MKL methods seek the combined kernel that performs best over every training example, sacrificing performance in some areas to seek a global optimum. Localized kernel learning (LKL) overcomes this limitation by allowing the training algorithm to match a component kernel to the examples that can exploit it best. Several approaches to the localized kernel learning problem have been explored in the last several years. We unify many of these approaches under one simple system and design a new algorithm with improved performance. We also develop enhanced versions of existing algorithms, with an eye on scalability and performance.

Machine Learning
Probability-confidence-kernel-based localized multiple kernel learning with l p norm

Yina Han,Guizhong Liu

DOI: https://doi.org/10.1109/TSMCB.2011.2179291

2012-01-01

Abstract:Localized multiple kernel learning (LMKL) is an attractive strategy for combining multiple heterogeneous features in terms of their discriminative power for each individual sample. However, models excessively fitting to a specific sample would obstacle the extension to unseen data, while a more general form is often insufficient for diverse locality characterization. Hence, both learning sample-specific local models for each training datum and extending the learned models to unseen test data should be equally addressed in designing LMKL algorithm. In this paper, for an integrative solution, we propose a probability confidence kernel (PCK), which measures per-sample similarity with respect to probabilistic-prediction-based class attribute: The class attribute similarity complements the spatial-similarity-based base kernels for more reasonable locality characterization, and the predefined form of involved class probability density function facilitates the extension to the whole input space and ensures its statistical meaning. Incorporating PCK into support-vectormachine-based LMKL framework, we propose a new PCK-LMKL with arbitrary l p-norm constraint implied in the definition of PCKs, where both the parameters in PCK and the final classifier can be efficiently optimized in a joint manner. Evaluations of PCK-LMKL on both benchmark machine learning data sets (ten University of California Irvine (UCI) data sets) and challenging computer vision data sets (15-scene data set and Caltech-101 data set) have shown to achieve state-of-the-art performances. © 2012 IEEE.

A Majorization-Minimization Approach to Lq Norm Multiple Kernel Learning

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

ℓq-Norm Sample-Adaptive Multiple Kernel Learning

A Multiple Kernel Learning Model Based on P-Norm

Efficient sparse generalized multiple kernel learning

Training Lp Norm Multiple Kernel Learning in the Primal

Efficient Multiple Kernel Learning Algorithms Using Low-Rank Representation.

Selective multiple kernel learning for classification with ensemble strategy.

Matrix-Regularized Multiple Kernel Learning Via &Lt;inline-Formula&gt; &Lt;tex-Math Notation="latex"&gt;$(r,~p)$ &Lt;/tex-Math&gt; &Lt;/inline-Formula&gt; Norms

Unsupervised Multiple Kernel Learning

Multiple Kernel Learning Method Using MRMR Criterion and Kernel Alignment

Kernel Least-Mean Mixed-Norm Algorithm

A Primal Method for Multiple Kernel Learning

Absent Multiple Kernel Learning Algorithms.

Multiple Empirical Kernel Learning with Locality Preserving Constraint

Joint Learning of Distance Metric and Kernel Classifier via Multiple Kernel Learning.

Multiple Kernel Subspace Clustering with Local Structural Graph and Low-Rank Consensus Kernel Learning

Reduced Multiple Empirical Kernel Learning Machine

L2-norm multiple kernel learning and its application to biomedical data fusion

A Unified View of Localized Kernel Learning

Probability-confidence-kernel-based localized multiple kernel learning with l p norm

Matrix-Regularized Multiple Kernel Learning Via &Lt;inline-Formula> &Lt;tex-Math Notation="latex">$(r,~p)$ &Lt;/tex-Math> &Lt;/inline-Formula> Norms