Least Squares Support Vector Machines

B. Moor,J. Brabanter,T. V. Gestel,J. Suykens,J. Vandewalle

DOI: https://doi.org/10.1142/9789812776655

2002-11-14

Abstract:Support Vector Machines Basic Methods of Least Squares Support Vector Machines Bayesian Inference for LS-SVM Models Robustness Large Scale Problems LS-SVM for Unsupervised Learning LS-SVM for Recurrent Networks and Control.

Mathematics,Computer Science

What problem does this paper attempt to address?

Least Squares Support Vector Machines Based on Support Vector Degrees

Lijuan Li,Youfeng Li,Hongye Su,Jian Chu

DOI: https://doi.org/10.1007/11816157_160

2006-01-01

Abstract:A modified least squares support vector machines (LS-SVM) approach, which treats the training data points differently according to their different degrees of importance, is proposed in this paper. On each data point, a support vector degree is defined and it is associated with the corresponding absolute value of Lagrange multiplier. The experiment of identification of pH neutralization process with polluted measuring data is shown in this paper and the result indicates that the method is effective in identification of nonlinear system. By contrast with the basic LS-SVM, the result also shows the priority of the presented new algorithm.
New Predictive Control Algorithms Based on Least Squares Support Vector Machines

Liu Bin,Su Hong-ye,Chu Jian

DOI: https://doi.org/10.1007/s11768-004-0041-7

2005-01-01

Journal of Zhejiang University SCIENCE A

Abstract:Used for industrial process with different degree of nonlinearity, the two predictive control algorithms presented in this paper are based on Least Squares Support Vector Machines (LS-SVM) model. For the weakly nonlinear system, the system model is built by using LS-SVM with linear kernel function, and then the obtained linear LS-SVM model is transformed into linear input-output relation of the controlled system. However, for the strongly nonlinear system, the off-line model of the controlled system is built by using LS-SVM with Radial Basis Function (RBF) kernel. The obtained nonlinear LS-SVM model is linearized at each sampling instant of system running, after which the on-line linear input-output model of the system is built. Based on the obtained linear input-output model, the Generalized Predictive Control (GPC) algorithm is employed to implement predictive control for the controlled plant in both algorithms. The simulation results after the presented algorithms were implemented in two different industrial processes model; respectively revealed the effectiveness and merit of both algorithms.
Sparse Representation Based on Projection Method in Online Least Squares Support Vector Machines

Lijuan Li,Hongye Su,Jian Chu

DOI: https://doi.org/10.1007/s11768-009-7026-5

2009-01-01

Abstract:A sparse approximation algorithm based on projection is presented in this paper in order to overcome the limitation of the non-sparsity of least squares support vector machines (LS-SVM). The new inputs are projected into the subspace spanned by previous basis vectors (BV) and those inputs whose squared distance from the subspace is higher than a threshold are added in the BV set, while others are rejected. This consequently results in the sparse approximation. In addition, a recursive approach to deleting an exiting vector in the BV set is proposed. Then the online LS-SVM, sparse approximation and BV removal are combined to produce the sparse online LS-SVM algorithm that can control the size of memory irrespective of the processed data size. The suggested algorithm is applied in the online modeling of a pH neutralizing process and the isomerization plant of a refinery, respectively. The detailed comparison of computing time and precision is also given between the suggested algorithm and the nonsparse one. The results show that the proposed algorithm greatly improves the sparsity just with little cost of precision.
Least squares support vector machine based partially linear model identification

You-Feng Li,Li-Juan Li,Hong-Ye Su,Jian Chu

DOI: https://doi.org/10.1007/11816157_94

2006-01-01

Abstract:A nonlinear identification method was proposed for a class of partially linear models (PLM) which consist of a linear component summed with a nonlinear component in nonlinear ARX form. The method extends the standard least squares support vector machine (LSSVM) by replacing the equality constraint in the standard LSSVM with a PLM model. To guarantee the uniqueness of the linear coefficients, we imposed an additional explicit constraint on the feature map instead of an implicit constraint on the regressor vectors. Therefore the resulting PLM is a generalized version of the original one. Two examples show the effectiveness of the presented method.
Fuzzy Time Series Analysis Method Based on Least Squares Support Vector Machines

SHEN Bin,YAO Min,YI Wen-sheng

DOI: https://doi.org/10.3785/j.issn.1008-973x.2005.08.010

2005-01-01

Abstract:Aiming at the problem of low precision of traditional fuzzy time series (FTS) analysis methods, this work proposed a new method based on least squares support vector machines (LS-SVM), which is an efficient tool for pattern recognition and regression estimation. This method includes two parts. In the FTS processing part, establish heuristic rules and fuzzy variables and determine the universe of discourse, fuzzy sets and membership degree functions, then fuzzify the history data. The LS-SVM processing part uses LS-SVM instead of traditional fuzzy relationship computation to analyze fuzzy data, and produces the final results after defuzzification. Comparing with traditional FTS analysis methods, this new method can obtain higher accuracy and good generalization quality.
Using Svm To Model And Control Nonlinear Dynamical Systems

Haoran Zhang,Xiaodong Wang,Changjiang Zhang,Jin Wang

2006-01-01

Abstract:This paper firstly provides an short introduction to least square support vector machine (LSSVM), a new class of kernel-based techniques introduced in statistical learning theory and structural risk minimization, then use it to model and control nonlinear systems, in which nonlinear predictive control framework is presented. The predictive control law is derived by a new stochastic search optimization algorithm. Simulation experiments are performed and indicate that the proposed method provides satisfactory performance with excellent generalization property.
Sparse Least Squares Support Vector Machine for Function Estimation

Liang-zhi Gan,Hai-kuan Liu,You-xian Sun

DOI: https://doi.org/10.1007/11759966_149

2007-01-01

Abstract:Sparse least squares support vector machine (SLS-SVM) for regression is proposed to solve the problem of regression for large sample data. The samples are mapped into Reproducing Kernel Hilbert Space (RKHS) and span a subspace there. Then we could find the basis of the subspace. The basis can represent all the samples linearly. So we can get the least squares support vector machine by solving a small equations set. A numerical example is used to illustrate that this approach can be used to fit nonlinear models for large data set. Being compared with common least squares support vector machine, this method can find sparse solution without any pruning or surgeon, and the computing speed is much faster because the final result is found by solving a small-scale equations set.
Modeling method of least squares support vector regression based on vector base learning

CHEN Ai-jun,SONG Zhi-huan,LI Ping

DOI: https://doi.org/10.3969/j.issn.1000-8152.2007.01.001

2007-01-01

Abstract:To achieve a sparse solution for least squares support vector regression (LS-SVM), an algorithm called vector base learning (VBL) is proposed in this paper. Firstly, the concepts of base vector (BV), base vector set (BVS) and vector space are introduced. By calculating the angle between the new sample vector and the vector space, the criteria for determining whether the measurement vector is one of the BVS is then derived. This determination is carried out on-line for the coming new samples. This makes the solutions of LS-SVM having the feature of sparsity. To improve the modeling speed of LS-SVM, a recursive algorithm of increased memory mode for VBL algorithm is also proposed. Finally, simulation analysis and the modeling of a typical plant for water treatment clearly illustrated the validity and feasibility of the presented method.
Least Square Generalized Support Vector Machines for Regression

孙宗海,孙优贤

DOI: https://doi.org/10.3321/j.issn:1000-6788.2004.07.013

2004-01-01

Systems Engineering - Theory & Practice

Abstract:Least square generalized support vector machines (LS__GSVMs) are applied to regression estimation. LS__GSVMs' kernel functions have no or few limits when they are compared with standard support vector machines (SVMs). We give a presentation of quadratic programming (QP) problem for the LS__GSVMs. In order to solve the QP problem, we apply the combination of the gradient projection and successive overrelaxation (SOR) based on the matrix splitting. That is, we train the LS__GSVMs with above algorithm. Because SOR handles one point at a time, it can process very large datasets that need not reside in memory.
Least Squares Support Vector Machine Algorithm

GU Yanping,ZHAO Wenjie,WU Zhansong

DOI: https://doi.org/10.16511/j.cnki.qhdxxb.2010.07.035

2010-01-01

Abstract:The least squares support vector machine(LS-SVM) has been widely used in various fields for artificial intelligence.This paper describes the basic LS-SVM algorithm.The influence of the model parameters on the model precision,complexity and computing time are analyzed using statistical learning theory and numerical examples.The conclusions give a good guide for determining the model parameters.An improved LS-SVM algorithm is developed with the results showing that the improved algorithm having better overall performance and better efficiency for wider application.
M-estimator Based Support Vector Machine and Its Application

BAO Xin,DAI Liankui

DOI: https://doi.org/10.3321/j.issn:0438-1157.2009.07.021

2009-01-01

Abstract:Validity of training samples is important to regression model,however,outliers in training sample sets are hard to avoid,which will disturb the model accuracy.In order to overcome the influence of outliers to regression model,a new M-estimator based support vector machine(M-SVM) was introduced.It replaced the sum of square residuals in least squares support vector machine(LS-SVM) with objective function of M-estimator.An iterative modeling algorithm of the M-SVM was proposed.The M-SVM was used in a low-dimension data regression problem with outliers and in the NIR spectral analysis of gasoline research octane number.Experimental results showed that the M-SVM was more robust and accurate than other SVMs.
Sparse Least Absolute Deviation Support Vector Machine

Kang-Mo Jung,

DOI: https://doi.org/10.37727/jkdas.2023.25.5.1701

2023-10-31

The Korean Data Analysis Society

Abstract:The support vector machine solves a quadratic programming problem with linear inequality and equality constraints. However, it is not trivial to solve the quadratic problem. The least squares support vector machine(LS-SVM) solves a linear system by equality constraints instead of inequality constraints. LS-SVM is a popular method in regression and classification problems, because it effectively solves simple linear systems. There are two issues with the LS-SVM solution : the lack of robustness to outliers and the absence of sparseness. In this paper, we propose a sparse and robust support vector machine for regression problems using the least absolute deviation support vector machine (LAD-SVM) and recursive reduced LS-SVM (RR-LS-SVM). The split-Bregman iteration gives the exact solution for the LAD-SVM problem, while RR-LS-SVM gives a sparse solution with a much smaller number of all support vectors. Numerical experiments with simulation and benchmark data demonstrate that the proposed algorithm can achieve comparable performance to other methods in terms of robustness and sparseness.
Asymmetric Least Squares Support Vector Machine Classifiers

Xiaolin Huang,Lei Shi,Johan A. K. Suykens

DOI: https://doi.org/10.1016/j.csda.2013.09.015

IF: 2.035

2014-01-01

Computational Statistics & Data Analysis

Abstract:In the field of classification, the support vector machine (SVM) pursues a large margin between two classes. The margin is usually measured by the minimal distance between two sets, which is related to the hinge loss or the squared hinge loss. However, the minimal value is sensitive to noise and unstable to re-sampling. To overcome this weak point, the expectile value is considered to measure the margin between classes instead of the minimal value. Motivated by the relation between the expectile value and the asymmetric squared loss, an asymmetric least squares SVM (aLS-SVM) is proposed. The proposed aLS-SVM can also be regarded as an extension to the LS-SVM and the L2-SVM. Theoretical analysis and numerical experiments on the aLS-SVM illustrate its insensitivity to noise around the boundary and its stability to re-sampling.
Dynamic least squares support vector machine

Fan Yugang,Li Ping,Song Zhihuan

DOI: https://doi.org/10.1109/WCICA.2006.1713313

2006-01-01

Abstract:Based on narrating the theory of least squares support vector machine (LS-SVM), dynamic LS-SVM (DLS-SVM) is presented in this paper. DLS-SVM is suitable for real time system recognition and time series prediction. Whenever a new example is obtained, the method gets rid of the first vector and replaces it with the new input vector. That is, this algorithm can adjust the model to track the dynamics of the nonlinear time-varying system. Time series prediction can be a very useful tool to forecast and to study the behavior of key process parameters in time. This creates the possibility to give early warnings of possible process malfunctioning. In this paper, DLS-SVM is applied to predict the concentration of 4-Carboxybenzaldchydc (4-CBA) in purified terephthalic acid (PTA) oxidation process. Results indicate that the proposed method is effective. © 2006 IEEE.
Robust least squares one-class support vector machine

Hong-Jie Xing,Li-Fei Li

DOI: https://doi.org/10.1016/j.patrec.2020.09.005

IF: 4.757

2020-10-01

Pattern Recognition Letters

Abstract:<p>In comparison with the conventional one-class support vector machine (OCSVM), least squares OCSVM (LS-OCSVM) can describe similarity between a new-coming sample and training set more accurately. However, LS-OCSVM is very sensitive to outliers in training set. The main reason lies that the values of square error function for outliers are relatively large, which makes LS-OCSVM put more emphasis on these outliers. To enhance the robustness of LS-OCSVM against outliers, a novel robust LS-OCSVM based on correntropy loss function is proposed. As a result, the unbounded convex square loss function of LS-OCSVM is substituted by a bounded nonconvex correntropy loss function. Experimental results on synthetic and benchmark data sets show that robust LS-OCSVM possesses better anti-outlier and generalization abilities in comparison with its related approaches.</p>

computer science, artificial intelligence
Minimal Support Vector Machine

Shuai Zheng,Chris Ding

DOI: https://doi.org/10.48550/arXiv.1804.02370

2018-04-07

Abstract:Support Vector Machine (SVM) is an efficient classification approach, which finds a hyperplane to separate data from different classes. This hyperplane is determined by support vectors. In existing SVM formulations, the objective function uses L2 norm or L1 norm on slack variables. The number of support vectors is a measure of generalization errors. In this work, we propose a Minimal SVM, which uses L0.5 norm on slack variables. The result model further reduces the number of support vectors and increases the classification performance.

Machine Learning
Randomized Kernel Methods for Least-Squares Support Vector Machines

M. Andrecut

DOI: https://doi.org/10.1142/S0129183117500152

2017-03-23

Abstract:The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that they provide good accuracy and reliable scaling for multi-class classification problems with relatively large data sets. Also, we present several numerical experiments that illustrate the practical applicability of the proposed methods.

Machine Learning,Data Analysis, Statistics and Probability
Sparse Least Squares Low Rank Kernel Machines

Di Xu,Manjing Fang,Xia Hong,Junbin Gao

DOI: https://doi.org/10.48550/arXiv.1901.10098

2019-10-19

Abstract:A general framework of least squares support vector machine with low rank kernels, referred to as LR-LSSVM, is introduced in this paper. The special structure of low rank kernels with a controlled model size brings sparsity as well as computational efficiency to the proposed model. Meanwhile, a two-step optimization algorithm with three different criteria is proposed and various experiments are carried out using the example of the so-call robust RBF kernel to validate the model. The experiment results show that the performance of the proposed algorithm is comparable or superior to several existing kernel machines.

Machine Learning,Artificial Intelligence
Multi-output least-squares support vector regression machines

Shuo Xu,Xin An,Xiaodong Qiao,Lijun Zhu,Lin Li

DOI: https://doi.org/10.1016/j.patrec.2013.01.015

IF: 4.757

2013-07-01

Pattern Recognition Letters

Abstract:Multi-output regression aims at learning a mapping from a multivariate input feature space to a multivariate output space. Despite its potential usefulness, the standard formulation of the least-squares support vector regression machine (LS-SVR) cannot cope with the multi-output case. The usual procedure is to train multiple independent LS-SVR, thus disregarding the underlying (potentially nonlinear) cross relatedness among different outputs. To address this problem, inspired by the multi-task learning methods, this study proposes a novel approach, Multi-output LS-SVR (MLS-SVR), in multi-output setting. Furthermore, a more efficient training algorithm is also given. Finally, extensive experimental results validate the effectiveness of the proposed approach.

computer science, artificial intelligence
Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression

Xin Xie,Chao Chen,Zhijian Chen

DOI: https://doi.org/10.2991/icmmcce-15.2015.503

2015-01-01

Abstract:Linear support vector regression (SVR) is a popular machine learning algorithm. However, as the amount of data increases, the learning procedure of SVR becomes time consuming. In this paper, we propose a mini-batch quasi-Newton optimization algorithm to speed up the training process of linear SVR. The main idea of the proposed optimization method is to use a small set of training data to estimate the first and second order gradient information and incorporate them into the framework of the popular limited memory BFGS quasi-Newton algorithm. Some modifications have been made to the generation of correction pairs of the BFGS algorithm in order to avoid the source of noise. Experimental results show that the proposed method outperforms some state-of-art methods in both training time and generalization ability.

Least Squares Support Vector Machines

Least Squares Support Vector Machines Based on Support Vector Degrees

New Predictive Control Algorithms Based on Least Squares Support Vector Machines

Sparse Representation Based on Projection Method in Online Least Squares Support Vector Machines

Least squares support vector machine based partially linear model identification

Fuzzy Time Series Analysis Method Based on Least Squares Support Vector Machines

Using Svm To Model And Control Nonlinear Dynamical Systems

Sparse Least Squares Support Vector Machine for Function Estimation

Modeling method of least squares support vector regression based on vector base learning

Least Square Generalized Support Vector Machines for Regression

Least Squares Support Vector Machine Algorithm

M-estimator Based Support Vector Machine and Its Application

Sparse Least Absolute Deviation Support Vector Machine

Asymmetric Least Squares Support Vector Machine Classifiers

Dynamic least squares support vector machine

Robust least squares one-class support vector machine

Minimal Support Vector Machine

Randomized Kernel Methods for Least-Squares Support Vector Machines

Sparse Least Squares Low Rank Kernel Machines

Multi-output least-squares support vector regression machines

Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression