On a New Kth-Order Quadratic Learning Algorithm

SH Tan,XH Zhang,Q Zhu

DOI: https://doi.org/10.1109/81.554342

1997-01-01

IEEE Transactions on Circuits and Systems I Fundamental Theory and Applications

Abstract:The authors develop a new kth-order quadratic learning algorithm, and compare it with the conventional least squares algorithm. It appears that when the order k is appropriately chosen, the algorithm can improve the learning efficiency

What problem does this paper attempt to address?

A high-order recursive quadratic learning algorithm

Qi Zhu,Shaohua Tan,Ying Qiao

DOI: https://doi.org/10.1007/11428831_12

2005-01-01

Abstract:A k-order Recursive Quadratic learning algorithm is proposed and its features are described in detail in this paper. Simulations are carried out to illustrate the efficiency and effectiveness of this new algorithm by comparing the results with both the projection algorithm and the conventional least squares algorithm.
A robust high-order recursive quadratic algorithm for linear-in-the-parameters models

Qi Zhu,Shaohua Tan,Ying Qiao

2006-01-01

Abstract:In this paper a robust k-order Recursive Quadratic algorithm for a large class of linear-in-the-parametes models is introduced to enhance Recursive Quadratic algorithm's robustness. The features of the algorithm are discussed and the convergence of the algorithm has been proven in this paper. Several examples are included to demonstrate the efficiency by comparing the result with the conventional least square algorithm and the effectiveness of the robust Recursive Quadratic algorithm.
Incremental batch learning with support vector machines

Yangguang Liu,Qinming He,Qi Chen

DOI: https://doi.org/10.1109/wcica.2004.1340997

2004-01-01

Abstract:Support Vector Machines (SVMs) have become a popular tool for learning with large amounts of high dimensional data. But it may sometimes be preferable to learn incrementally from previous SVM results, as SVMs which involve the solution of a quadratic programming problem suffer from the problem of large memory requirement and CPU time when trained in batch mode on large data sets. And the SVMs may be used in online learning setting. We proposed an approach for incremental batch learning with Support Vector Machines for classification and regression, and define the normal solution of the incremental batch learning with SVMs as the solution minimizing a given positive-definite quadratic form in the coordinates of the difference vector between the normal vectors at the (k-1)-th and k-th incremental batch step and discuss the relation to standard SVM. It is shown that concept learned at last step will not change if new data satisfy separable condition and empirical evidence is given to prove that this approach can effectively deal with changes in the target concept that are results of the incremental learning setting according to three evaluation criteria: stability, improvement and recoverability.
An Incremental Updating Method for Support Vector Machines

Yangguang Liu,Qi Chen,Yongchuan Tang,Qinming He

DOI: https://doi.org/10.1007/978-3-540-24655-8_45

2004-01-01

Abstract:Support Vector Machines (SVMs) have become a popular tool for learning with large amounts of high dimensional data. But it may sometimes be preferable to learn incrementally from previous SVM results, as SVMs which involve the solution of a quadratic programming problem suffer from the problem of large memory requirement and CPU time when trained in batch mode on large data sets. And the SVMs may be used in online learning setting. In this paper an approach for incremental learning with Support Vector Machines is presented. We define the normal solution of the incremental learning for SVMs which is defined as the solution minimizing a given positive-definite quadratic form in the coordinates of the difference vector between the normal vectors at the (k-1)-th and k-th incremental step and discuss the relation to standard SVM. It was shown that concept learned at last step will not change if new data satisfy separable condition and empirical evidence is given to prove that this approach can effectively deal with changes in the target concept that are results of the incremental learning setting according to three evaluation criteria: stability, improvement and recoverability.
An On-Line Learning Approach with Support Vector Dormain Classifier

Ying-Gang Zhao,Shuo-Ping Wang,Yang-Guang Liu,Qin-Ming He

DOI: https://doi.org/10.1109/icmlc.2006.258578

2006-01-01

Abstract:As one kind of one-class classifier, support vector domain classifier (SVDC) has worked well for the batch model learning problems. But with real-world database increase in size, there is a need to scale up inductive learning algorithm to handle more training data. On-line learning technique is one possible solution to the scalability problem, where data is processed in parts, and the result combined so as to use less memory. This paper presented an on-line learning algorithm based on SVDC, and the basic idea of the proposed algorithm is to obtain the initial target concepts using SVDC during the training phase and then update these target concepts by an updating model. Different from the existed on-line learning approaches, in our algorithm, the model updating procedure equals to solve a quadratic programming (QP) problem, and the updated model still owns the property of spars solution. Compared with other existed on-line learning algorithms, the inverse procedure of our algorithm (i.e. decreasing learning) is easy to conduct without extra computation
A High-Order Recursive Quadratic Algorithm for Linear-in-the-Parameter Models

Qi Zhu,Ying Qiao,Shaohua Tan

DOI: https://doi.org/10.1142/S0218126608004459

2011-01-01

Journal of Circuits Systems and Computers

Abstract:In this paper, a high-order Recursive Quadratic (RQ) algorithm is introduced, and the features of this algorithm are thoroughly studied and illustrated. In addition, a robust RQ algorithm is also developed in the presence of general bounded noisy data to enhance model robustness. A numerical example is included to demonstrate the efficiency of RQ algorithms by comparing the results with both the projection algorithm and the conventional recursive least squares algorithm. Also some simulations are carried out to illustrate the effectiveness of the RQ and robust RQ algorithms.
Online Efficient Learning with Quantized KLMS and L<inf>1</inf> Regularization

Badong Chen,Songlin Zhao,Sohan Seth,José C. Prı́ncipe

DOI: https://doi.org/10.1109/ijcnn.2012.6252455

2012-01-01

Abstract:In a recent work, we have proposed the quantized kernel least mean square (QKLMS) algorithm, which is quite effective in online learning sequentially a nonlinear mapping with a slowly growing radial basis function (RBF) structure. In this paper, in order to further reduce the network size, we propose a sparse QKLMS algorithm, which is derived by adding a sparsity inducing l 1 norm penalty of the coefficients to the squared error cost. Simulation examples show that the new algorithm works efficiently, and results in a much sparser network while preserving a desirable performance.
Adaptive quasiconformal kernel discriminant analysis

Pan Jeng-Shyang,Li Jun-Bao,Lu Zhe-Ming

DOI: https://doi.org/10.1016/j.neucom.2007.10.003

IF: 6

2008-01-01

Neurocomputing

Abstract:Kernel discriminant analysis (KDA) is effective to extract nonlinear discriminative features of input samples using the kernel trick. However, the conventional KDA algorithm endures the kernel selection which has significant impact on the performances of KDA. In order to overcome this limitation, a novel nonlinear feature extraction method called adaptive quasiconformal kernel discriminant analysis (AQKDA) is proposed in this paper. AQKDA maps the data from the original input space to the high dimensional kernel space using a quasiconformal kernel. The adaptive parameters of the quasiconformal kernel are automatically calculated through optimizing an objective function designed for measuring the class separability of data in the feature space. Consequently, the nonlinear features extracted by AQKDA have the larger class separability compared with KDA. Experimental results on the two real-world datasets demonstrate the effectiveness of the proposed method. © 2007 Elsevier B.V. All rights reserved.
A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

Albert S. Berahas,Frank E. Curtis,Michael J. O'Neill,Daniel P. Robinson

DOI: https://doi.org/10.48550/arXiv.2106.13015

2023-03-16

Abstract:A sequential quadratic optimization algorithm is proposed for solving smooth nonlinear equality constrained optimization problems in which the objective function is defined by an expectation of a stochastic function. The algorithmic structure of the proposed method is based on a step decomposition strategy that is known in the literature to be widely effective in practice, wherein each search direction is computed as the sum of a normal step (toward linearized feasibility) and a tangential step (toward objective decrease in the null space of the constraint Jacobian). However, the proposed method is unique from others in the literature in that it both allows the use of stochastic objective gradient estimates and possesses convergence guarantees even in the setting in which the constraint Jacobians may be rank deficient. The results of numerical experiments demonstrate that the algorithm offers superior performance when compared to popular alternatives.

Optimization and Control,Machine Learning
Online Efficient Learning with Quantized Klms and L-1 Regularization

Badong Chen,Songlin Zhao,Sohan Seth,Jose C. Principe

2012-01-01

Abstract:In a recent work, we have proposed the quantized kernel least mean square (QKLMS) algorithm, which is quite effective in online learning sequentially a nonlinear mapping with a slowly growing radial basis function (RBF) structure. In this paper, in order to further reduce the network size, we propose a sparse QKLMS algorithm, which is derived by adding a sparsity inducing l(1) norm penalty of the coefficients to the squared error cost. Simulation examples show that the new algorithm works efficiently, and results in a much sparser network while preserving a desirable performance.
A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

Yuli He,Yong Wang

DOI: https://doi.org/10.1109/ICFDA58234.2023.10153257

2023-01-01

Abstract:In an attempt to solve the problem of the difficult choice of step size for the fractional order gradient descent method, this article proposes a novel fractional order speedest gradient descent method for the quadratic loss function. Based on Riemann-Liouville definition, a fractional-order gradient descent method with an adaptive learning rate clamping mechanism is proposed. The method is capable of using historical learning rate and current fractional order gradient information to adaptively change the learning rate and accelerate the convergence of the algorithm. By selecting proper hyperparameters, the proposed method has a rapid convergence rate and high accuracy in numerical examples.
Kernel Least Mean Square Based on Conjugate Gradient.

Siyuan Peng,Zongze Wu,Wentao Ma,Badong Chen

DOI: https://doi.org/10.1109/icassp.2017.7952666

2017-01-01

Abstract:Kernel least mean square (KLMS) algorithm has been successfully applied in fields of adaptive filtering and online learning due to their ability to solve sequentially nonlinear problems by implicitly mapping the input signal to a high-dimensional reproducing kernel Hilbert space (RKHS). In this paper, we propose a novel adaptive algorithm called KLMS based on conjugate gradient (KLMS-CG), which uses the orthogonal search directions, instead of using the traditional steepest descent approach, to improve the convergence speed. Further, the quantized KLMS based on conjugate gradient (QKLMS-CG) is proposed to curb the growth of network. Simulation results indicate that the new algorithm can converge faster than the original KLMS while maintaining excellent accuracy.
Fast Second-Order Online Kernel Learning through Incremental Matrix Sketching and Decomposition

Dongxie Wen,Xiao Zhang,Zhewei Wei

2024-10-15

Abstract:Online Kernel Learning (OKL) has attracted considerable research interest due to its promising predictive performance in streaming environments. Second-order approaches are particularly appealing for OKL as they often offer substantial improvements in regret guarantees. However, existing second-order OKL approaches suffer from at least quadratic time complexity with respect to the pre-set budget, rendering them unsuitable for meeting the real-time demands of large-scale streaming recommender systems. The singular value decomposition required to obtain explicit feature mapping is also computationally expensive due to the complete decomposition process. Moreover, the absence of incremental updates to manage approximate kernel space causes these algorithms to perform poorly in adversarial environments and real-world streaming recommendation datasets. To address these issues, we propose FORKS, a fast incremental matrix sketching and decomposition approach tailored for second-order OKL. FORKS constructs an incremental maintenance paradigm for second-order kernelized gradient descent, which includes incremental matrix sketching for kernel approximation and incremental matrix decomposition for explicit feature mapping construction. Theoretical analysis demonstrates that FORKS achieves a logarithmic regret guarantee on par with other second-order approaches while maintaining a linear time complexity w.r.t. the budget, significantly enhancing efficiency over existing approaches. We validate the performance of FORKS through extensive experiments conducted on real-world streaming recommendation datasets, demonstrating its superior scalability and robustness against adversarial attacks.

Machine Learning
QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation

Chenhui Xu,Xinyao Wang,Fuxun Yu,Jinjun Xiong,Xiang Chen

2024-05-09

Abstract:Machine learning is evolving towards high-order models that necessitate pre-training on extensive datasets, a process associated with significant overheads. Traditional models, despite having pre-trained weights, are becoming obsolete due to architectural differences that obstruct the effective transfer and initialization of these weights. To address these challenges, we introduce a novel framework, QuadraNet V2, which leverages quadratic neural networks to create efficient and sustainable high-order learning models. Our method initializes the primary term of the quadratic neuron using a standard neural network, while the quadratic term is employed to adaptively enhance the learning of data non-linearity or shifts. This integration of pre-trained primary terms with quadratic terms, which possess advanced modeling capabilities, significantly augments the information characterization capacity of the high-order network. By utilizing existing pre-trained weights, QuadraNet V2 reduces the required GPU hours for training by 90\% to 98.4\% compared to training from scratch, demonstrating both efficiency and effectiveness.

Machine Learning,Artificial Intelligence
Sequential Updating Algorithm for Extracting the Basis of Karhunen Loeve Transformation

YY Qu,NN Zheng,CH Li,ZJ Yuan

DOI: https://doi.org/10.1109/icip.2004.1421356

2004-01-01

Abstract:Karhunen-Loeve transformation (KLT) is a popular method for dimensional reduction and feature extraction in image analysis, signal processing, automatic control systems, and so on, while the drawback of the KLT is an expensive computation. In this paper, we propose a novel updating algorithm for KLT, the rank-k updating algorithm, which has advantages especially for image sequences: it is faster than batch algorithm, it can handle the dynamic database, and it does not save the entire database. Furthermore it makes the active learning and recognition possible in computer vision. Finally we analyze the computational complexity and error of the algorithm. We also show its application in face analysis. The experimental results demonstrate the efficiency of our algorithm.
An Accelerated Proximal Gradient-Based Algorithm for Quadratic Programming

Jia Wang,Ying Yang

DOI: https://doi.org/10.1109/lcsys.2022.3154597

2022-01-01

IEEE Control Systems Letters

Abstract:In this letter, an accelerated quadratic programming (QP) algorithm is proposed based on the proximal gradient method. The algorithm can achieve convergence rate $O{(}1/p^{\alpha }{)}$ , where $p$ is the iteration number and $\alpha $ is a given positive integer. The proposed algorithm improves the convergence rate of existing algorithms that achieve $O{(}1/p^{2}{)}$ . The key idea is that iterative parameters are selected from a group of specific high order polynomial equations. The performance of the proposed algorithm is assessed on the randomly generated model predictive control optimization problems. The effectiveness of our algorithm is verified by the numerical experiments.
Kernel-Free Quadratic Surface Regression for Multi-Class Classification

Changlin Wang,Zhixia Yang,Junyou Ye,Xue Yang

DOI: https://doi.org/10.3390/e25071103

2023-07-24

Abstract:For multi-class classification problems, a new kernel-free nonlinear classifier is presented, called the hard quadratic surface least squares regression (HQSLSR). It combines the benefits of the least squares loss function and quadratic kernel-free trick. The optimization problem of HQSLSR is convex and unconstrained, making it easy to solve. Further, to improve the generalization ability of HQSLSR, a softened version (SQSLSR) is proposed by introducing an ε-dragging technique, which can enlarge the between-class distance. The optimization problem of SQSLSR is solved by designing an alteration iteration algorithm. The convergence, interpretability and computational complexity of our methods are addressed in a theoretical analysis. The visualization results on five artificial datasets demonstrate that the obtained regression function in each category has geometric diversity and the advantage of the ε-dragging technique. Furthermore, experimental results on benchmark datasets show that our methods perform comparably to some state-of-the-art classifiers.
Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

John Chiang

DOI: https://doi.org/10.48550/arXiv.2209.03282

2023-03-29

Abstract:It might be inadequate for the line search technique for Newton's method to use only one floating point number. A column vector of the same size as the gradient might be better than a mere float number to accelerate each of the gradient elements with different rates. Moreover, a square matrix of the same order as the Hessian matrix might be helpful to correct the Hessian matrix. Chiang applied something between a column vector and a square matrix, namely a diagonal matrix, to accelerate the gradient and further proposed a faster gradient variant called quadratic gradient. In this paper, we present a new way to build a new version of the quadratic gradient. This new quadratic gradient doesn't satisfy the convergence conditions of the fixed Hessian Newton's method. However, experimental results show that it sometimes has a better performance than the original one in convergence rate. Also, Chiang speculates that there might be a relation between the Hessian matrix and the learning rate for the first-order gradient descent method. We prove that the floating number $\frac{1}{\epsilon + \max \{| \lambda_i | \}}$ can be a good learning rate of the gradient methods, where $\epsilon$ is a number to avoid division by zero and $\lambda_i$ the eigenvalues of the Hessian matrix.

Optimization and Control,Machine Learning
Quaternion kernel recursive least-squares algorithm

Gang Wang,Jingci Qiao,Rui Xue,Bei Peng

DOI: https://doi.org/10.1016/j.sigpro.2020.107810

IF: 4.729

2021-01-01

Signal Processing

Abstract:<p>Various kernel-based algorithms have been successfully applied to nonlinear problems in adaptive filters over the last two decades. In this paper, we study a kernel recursive least squares (KRLS) algorithm in the quaternion domain. By the generalized Hamilton-real calculus method, we can apply the kernel trick to calculate the quaternion KRLS filter. In order to show the feasibility of the proposed algorithm, firstly we investigate the quaternion recursive least squares (QRLS) algorithm, and simulations show that the proposed QRLS algorithm has the same steady error as that of the closed-form solution; Secondly, we generalize the QRLS algorithm to the quaternion KRLS algorithm, theoretical analysis show the convergence, and simulations are described demonstrating the performance of the proposed algorithm.</p>

engineering, electrical & electronic
Krylov Subspace Approximation for Quadratic-Bilinear Differential System

Jun-Man Yang,Yao-Lin Jiang

DOI: https://doi.org/10.1080/00207721.2018.1480814

IF: 2.648

2018-01-01

International Journal of Systems Science

Abstract:Many nonlinear systems with nonlinearities of the form 1/(k + x), ex, xa, ln(x) can be converted into quadratic-bilinear differential algebraic equations (QBDAEs) by introducing new variables and operating some algebra computations. Previous researches claim that the first two generalised transfer functions are enough to capture the dynamical behaviours of the system. In this research, weattempt to use traditional Krylov subspace methods to obtain the reduced model of the original QBDAEs so that the input-output behaviours of the reduced system can approximate that of the initial system. Model order reduction method proposed in this paper can match higher moments of first two transfer functions. Moreover, it does not involve the computation of tensor, and the main subfunction of this method is Arnoldi process which is easy to realise. We also analyse the stability and matching property of the reduction process. In addition, the numerical cost is given. The theory result and numerical result show the validity of the proposed method and the small computation expense of this method.

On a New Kth-Order Quadratic Learning Algorithm

A high-order recursive quadratic learning algorithm

A robust high-order recursive quadratic algorithm for linear-in-the-parameters models

Incremental batch learning with support vector machines

An Incremental Updating Method for Support Vector Machines

An On-Line Learning Approach with Support Vector Dormain Classifier

A High-Order Recursive Quadratic Algorithm for Linear-in-the-Parameter Models

Online Efficient Learning with Quantized KLMS and L&lt;inf&gt;1&lt;/inf&gt; Regularization

Adaptive quasiconformal kernel discriminant analysis

A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

Online Efficient Learning with Quantized Klms and L-1 Regularization

A Novel Fractional Order Speedest Gradient Descent Method and its application<sup>*</sup>

Kernel Least Mean Square Based on Conjugate Gradient.

Fast Second-Order Online Kernel Learning through Incremental Matrix Sketching and Decomposition

QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation

Sequential Updating Algorithm for Extracting the Basis of Karhunen Loeve Transformation

An Accelerated Proximal Gradient-Based Algorithm for Quadratic Programming

Kernel-Free Quadratic Surface Regression for Multi-Class Classification

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

Quaternion kernel recursive least-squares algorithm

Krylov Subspace Approximation for Quadratic-Bilinear Differential System

Online Efficient Learning with Quantized KLMS and L<inf>1</inf> Regularization