Sparse Online Learning via Truncated Gradient.

John Langford, Lihong Li, Tong Zhang

2009-03-01

Abstract:We propose a general method called truncated gradient to induce sparsity in the weights of onlinelearning algorithms with convex loss functions. This method has several essential properties:

What problem does this paper attempt to address?

Quantum Algorithm for Sparse Online Learning with Truncated Gradient Descent

Debbie Lim,Yixian Qiu,Patrick Rebentrost,Qisheng Wang

2024-11-06

Abstract:Logistic regression, the Support Vector Machine (SVM), and least squares are well-studied methods in the statistical and computer science community, with various practical applications. High-dimensional data arriving on a real-time basis makes the design of online learning algorithms that produce sparse solutions essential. The seminal work of \hyperlink{cite.langford2009sparse}{Langford, Li, and Zhang (2009)} developed a method to obtain sparsity via truncated gradient descent, showing a near-optimal online regret bound. Based on this method, we develop a quantum sparse online learning algorithm for logistic regression, the SVM, and least squares. Given efficient quantum access to the inputs, we show that a quadratic speedup in the time complexity with respect to the dimension of the problem is achievable, while maintaining a regret of $O(1/\sqrt{T})$, where $T$ is the number of iterations.

Machine Learning,Quantum Physics
Online Kernel Learning with a Near Optimal Sparsity Bound

Lijun Zhang,Jinfeng Yi,Rong Jin,Ming Lin,Xiaofei He

2013-01-01

Abstract:In this work, we focus on Online Sparse Kernel Learning that aims to online learn a kernel classifier with a bounded number of support vectors. Although many online learning algorithms have been proposed to learn a sparse kernel classifier, most of them fail to bound the number of support vectors used by the final solution which is the average of the intermediate kernel classifiers generated by online algorithms. The key idea of the proposed algorithm is to measure the difficulty in correctly classifying a training example by the derivative of a smooth loss function, and give a more chance to a difficult example to be a support vector than an easy one via a sampling scheme. Our analysis shows that when the loss function is smooth, the proposed algorithm yields similar performance guarantee as the standard online learning algorithm but with a near optimal number of support vectors (up to a poly(ln T) factor). Our empirical study shows promising performance of the proposed algorithm compared to the state-of-the-art algorithms for online sparse kernel learning.
Screening for Sparse Online Learning

Jingwei Liang,Clarice Poon

DOI: https://doi.org/10.48550/arXiv.2101.06982

2022-11-11

Abstract:Sparsity promoting regularizers are widely used to impose low-complexity structure (e.g. l1-norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequence generated by iterative algorithms (such as proximal gradient descent) exhibit "finite activity identification", namely, they can identify the low-complexity structure in a finite number of iterations. However, most online algorithms (such as proximal stochastic gradient descent) do not have the property owing to the vanishing step-size and non-vanishing variance. In this paper, by combining with a screening rule, we show how to eliminate useless features of the iterates generated by online algorithms, and thereby enforce finite activity identification. One consequence is that when combined with any convergent online algorithm, sparsity properties imposed by the regularizer can be exploited for computational gains. Numerically, significant acceleration can be obtained.

Machine Learning,Optimization and Control
Learning with Non-Convex Truncated Losses by SGD

Yi Xu,Shenghuo Zhu,Sen Yang,Chi Zhang,Rong Jin,Tianbao Yang

DOI: https://doi.org/10.48550/arxiv.1805.07880

2018-01-01

Abstract:Learning with a {\it convex loss} function has been a dominating paradigm for many years. It remains an interesting question how non-convex loss functions help improve the generalization of learning with broad applicability. In this paper, we study a family of objective functions formed by truncating traditional loss functions, which is applicable to both shallow learning and deep learning. Truncating loss functions has potential to be less vulnerable and more robust to large noise in observations that could be adversarial. More importantly, it is a generic technique without assuming the knowledge of noise distribution. To justify non-convex learning with truncated losses, we establish excess risk bounds of empirical risk minimization based on truncated losses for heavy-tailed output, and statistical error of an approximate stationary point found by stochastic gradient descent (SGD) method. Our experiments for shallow and deep learning for regression with outliers, corrupted data and heavy-tailed noise further justify the proposed method.
Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Qing Tao,Gaowei Wu,Dejun Chu

DOI: https://doi.org/10.1109/tnnls.2017.2705429

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The truncated regular L-1-loss support vector machine can eliminate the excessive number of support vectors (SVs); thus, it has significant advantages in robustness and scalability. However, in this paper, we discover that the associated state-of-the-art solvers, such as difference convex algorithm and concave-convex procedure, not only have limited sparsity promoting property for general truncated losses especially the L-2-loss but also have poor scalability for large-scale problems. To circumvent these drawbacks, we present a general multistage scheme with explicit interpretation regarding SVs as well as outliers. In particular, we solve the general nonconvex truncated loss minimization through a sequence of associated convex subproblems, in which the outliers are removed in advance. The proposed algorithm can be regarded as a structural optimization attempt carefully considering sparsity imposed by the nonconvex truncated losses. We show that this general multistage algorithm offers sufficient sparsity especially for the truncated L-2-loss. To further improve the scalability, we propose a linear multistep algorithm by employing a single iteration of coordinate descent to monotonically decrease the objective function at each stage and a kernel algorithm by using the Karush-Kuhn-Tucker conditions to cheaply find most part of the outliers for the next stage. Comparison experiments demonstrate that our methods have superiority in sparsity as well as efficiency in scalability.
A Dual Perspective of Sparse and Robust Online Learning Algorithm

Min Tang,Boliang Sun,Guohui Li

DOI: https://doi.org/10.1145/2632856.2632922

2014-01-01

Abstract:In this paper, we propose a dual perspective of online learning algorithm, which concerns using a window method to achieve sparsity and robustness. It makes use of Fenchel conjugates and gradient ascent to perform online learning optimization process. The window method is an update strategy for the classifier. It consists of two bounds which related to the dual increase. The lower bound abandons the points which induce a smaller dual ascent while the upper bound constraints the dual increase generated by noise points to reduce their influence on the target boundary. Moreover, with the use of the window method, the prediction accuracy can be increased significantly. Detailed experiments on artificial and real world datasets verify the utility of the proposed approaches.
Gradient Sparsification for Communication-Efficient Distributed Optimization

Jianqiao Wangni,Jialei Wang,Ji Liu,Tong Zhang

DOI: https://doi.org/10.48550/arXiv.1710.09854

2017-10-27

Abstract:Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost we propose a convex optimization formulation to minimize the coding length of stochastic gradients. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for approximate solution, with theoretical guaranteed for sparseness. Experiments on $\ell_2$ regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

Machine Learning,Numerical Analysis
Variable Screening for Sparse Online Regression.

Jingwei Liang,Clarice Poon

DOI: https://doi.org/10.1080/10618600.2022.2099872

2022-01-01

Journal of Computational and Graphical Statistics

Abstract:Sparsity-promoting regularizers are widely used to impose low-complexity structure (e.g., l(1)-norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequence generated by iterative algorithms (such as proximal gradient descent) exhibit "finite activity identification" property, that is, they can identify the low-complexity structure of the solution in a finite number of iterations. However, many online algorithms (such as proximal stochastic gradient descent) do not have this property owing to the vanishing step-size and nonvanishing variance. In this article, by combining with a screening rule, we show how to eliminate useless features of the iterates generated by online algorithms, and thereby enforce finite sparsity identification. One advantage of our scheme is that when combined with any convergent online algorithm, sparsity properties imposed by the regularizer can be exploited to improve computational efficiency. Numerically, significant acceleration can be obtained.
Accelerating CNN Training by Sparsifying Activation Gradients

Ye Xucheng,Yang Jianlei,Dai Pengcheng,Chen Yiran,Zhao Weisheng

2019-01-01

Abstract: Gradients to activations get involved in most of the calculations during back propagation procedure of Convolution Neural Networks (CNNs) training. However, an important known observation is that the majority of these gradients are close to zero, imposing little impact on weights update. These gradients can be then pruned to achieve high gradient sparsity during CNNs training and reduce the computational cost. In particular, we randomly change a gradient to zero or a threshold value if the gradient is below the threshold which is determined by the statistical distribution of activation gradients. We also theoretically proved that the training convergence of the CNN model can be guaranteed when the above activation gradient sparsification method is applied. We evaluated our method on AlexNet, MobileNet, ResNet-{18, 34, 50, 101, 152} with CIFAR-{10, 100} and ImageNet datasets. Experimental results show that our method can substantially reduce the computational cost with negligible accuracy loss or even accuracy improvement. Finally, we analyzed the benefits that the sparsity of activation gradients introduced in detail.
Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via $\ell_1, \ell_0$, and transformed-$\ell_1$ Penalties

Thu Dinh,Jack Xin

DOI: https://doi.org/10.48550/arXiv.1812.05719

2018-12-13

Optimization and Control

Abstract:Sparsification of neural networks is one of the effective complexity reduction methods to improve efficiency and generalizability. We consider the problem of learning a one hidden layer convolutional neural network with ReLU activation function via gradient descent under sparsity promoting penalties. It is known that when the input data is Gaussian distributed, no-overlap networks (without penalties) in regression problems with ground truth can be learned in polynomial time at high probability. We propose a relaxed variable splitting method integrating thresholding and gradient descent to overcome the lack of non-smoothness in the loss function. The sparsity in network weight is realized during the optimization (training) process. We prove that under $\ell_1, \ell_0$; and transformed-$\ell_1$ penalties, no-overlap networks can be learned with high probability, and the iterative weights converge to a global limit which is a transformation of the true weight under a novel thresholding operation. Numerical experiments confirm theoretical findings, and compare the accuracy and sparsity trade-off among the penalties.
Training a neural netwok for data reduction and better generalization

Sylvain Sardy,Maxime van Cutsem,Xiaoyu Ma

2024-11-26

Abstract:The motivation for sparse learners is to compress the inputs (features) by selecting only the ones needed for good generalization. Linear models with LASSO-type regu- larization achieve this by setting the weights of irrelevant features to zero, effectively identifying and ignoring them. In artificial neural networks, this selective focus can be achieved by pruning the input layer. Given a cost function enhanced with a sparsity-promoting penalty, our proposal selects a regularization term $\lambda$ (without the use of cross-validation or a validation set) that creates a local minimum in the cost function at the origin where no features are selected. This local minimum acts as a baseline, meaning that if there is no strong enough signal to justify a feature inclusion, the local minimum remains at zero with a high prescribed probability. The method is flexible, applying to complex models ranging from shallow to deep artificial neural networks and supporting various cost functions and sparsity-promoting penalties. We empirically show a remarkable phase transition in the probability of retrieving the relevant features, as well as good generalization thanks to the choice of $\lambda$, the non-convex penalty and the optimization scheme developed. This approach can be seen as a form of compressed sensing for complex models, allowing us to distill high- dimensional data into a compact, interpretable subset of meaningful features.

Machine Learning
CONGO: Compressive Online Gradient Optimization

Jeremy Carleton,Prathik Vijaykumar,Divyanshu Saxena,Dheeraj Narasimha,Srinivas Shakkottai,Aditya Akella

2024-10-27

Abstract:We address the challenge of zeroth-order online convex optimization where the objective function's gradient exhibits sparsity, indicating that only a small number of dimensions possess non-zero gradients. Our aim is to leverage this sparsity to obtain useful estimates of the objective function's gradient even when the only information available is a limited number of function samples. Our motivation stems from the optimization of large-scale queueing networks that process time-sensitive jobs. Here, a job must be processed by potentially many queues in sequence to produce an output, and the service time at any queue is a function of the resources allocated to that queue. Since resources are costly, the end-to-end latency for jobs must be balanced with the overall cost of the resources used. While the number of queues is substantial, the latency function primarily reacts to resource changes in only a few, rendering the gradient sparse. We tackle this problem by introducing the Compressive Online Gradient Optimization framework which allows compressive sensing methods previously applied to stochastic optimization to achieve regret bounds with an optimal dependence on the time horizon without the full problem dimension appearing in the bound. For specific algorithms, we reduce the samples required per gradient estimate to scale with the gradient's sparsity factor rather than its full dimensionality. Numerical simulations and real-world microservices benchmarks demonstrate CONGO's superiority over gradient descent approaches that do not account for sparsity.

Machine Learning,Distributed, Parallel, and Cluster Computing,Optimization and Control
Noisy Truncated SGD: Optimization and Generalization

Yingxue Zhou,Xinyan Li,Arindam Banerjee

DOI: https://doi.org/10.48550/arXiv.2103.00075

2021-10-18

Abstract:Recent empirical work on stochastic gradient descent (SGD) applied to over-parameterized deep learning has shown that most gradient components over epochs are quite small. Inspired by such observations, we rigorously study properties of Truncated SGD (T-SGD), that truncates the majority of small gradient components to zeros. Considering non-convex optimization problems, we show that the convergence rate of T-SGD matches the order of vanilla SGD. We also establish the generalization error bound for T-SGD. Further, we propose Noisy Truncated SGD (NT-SGD), which adds Gaussian noise to the truncated gradients. We prove that NT-SGD has the same convergence rate as T-SGD for non-convex optimization problems. We demonstrate that with the help of noise, NT-SGD can provably escape from saddle points and requires less noise compared to previous related work. We also prove that NT-SGD achieves better generalization error bound compared to T-SGD because of the noise. Our generalization analysis is based on uniform stability and we show that additional noise in the gradient update can boost the stability. Our experiments on a variety of benchmark datasets (MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100) with various networks (VGG and ResNet) validate the theoretical properties of NT-SGD, i.e., NT-SGD matches the speed and accuracy of vanilla SGD while effectively working with sparse gradients, and can successfully escape poor local minima.

Machine Learning
Batch gradient training method with smoothing ℓ_ 0 regularization for feedforward neural networks

Huisheng Zhang,Yanli Tang,Xiaodong Liu

DOI: https://doi.org/10.1007/s00521-014-1730-x

2014-01-01

Neural Computing and Applications

Abstract:This paper considers the batch gradient method with the smoothing ℓ _0 regularization (BGSL0) for training and pruning feedforward neural networks. We show why BGSL0 can produce sparse weights, which are crucial for pruning networks. We prove both the weak convergence and strong convergence of BGSL0 under mild conditions. The decreasing monotonicity of the error functions during the training process is also obtained. Two examples are given to substantiate the theoretical analysis and to show the better sparsity of BGSL0 than three typical ℓ _p regularization methods.
Minimum Variance Unbiased N:M Sparsity for the Neural Gradients

Brian Chmiel,Itay Hubara,Ron Banner,Daniel Soudry

2024-06-10

Abstract:In deep learning, fine-grained N:M sparsity reduces the data footprint and bandwidth of a General Matrix multiply (GEMM) up to x2, and doubles throughput by skipping computation of zero values. So far, it was mainly only used to prune weights to accelerate the forward and backward phases. We examine how this method can be used also for the neural gradients (i.e., loss gradients with respect to the intermediate neural layer outputs). To this end, we first establish a tensor-level optimality criteria. Previous works aimed to minimize the mean-square-error (MSE) of each pruned block. We show that while minimization of the MSE works fine for pruning the weights and activations, it catastrophically fails for the neural gradients. Instead, we show that accurate pruning of the neural gradients requires an unbiased minimum-variance pruning mask. We design such specialized masks, and find that in most cases, 1:2 sparsity is sufficient for training, and 2:4 sparsity is usually enough when this is not the case. Further, we suggest combining several such methods together in order to potentially speed up training even more.

Machine Learning,Artificial Intelligence
Nonconvex Sparse Representation With Slowly Vanishing Gradient Regularizers

Eunwoo Kim,Minsik Lee,Songhwai Oh

DOI: https://doi.org/10.1109/access.2020.3009971

IF: 3.9

2020-01-01

IEEE Access

Abstract:Sparse representation has been widely used over the past decade in computer vision and signal processing to model a wide range of natural phenomena. For computational convenience and robustness against noises, the optimization problem for sparse representation is often relaxed using convex or nonconvex surrogates instead of using the $l_{0}$ -norm, the ideal sparsity penalty function. In this paper, we pose the following question for nonconvex sparsity-promoting surrogates: What is a good sparsity surrogate for general nonconvex systems? As an answer to this question, we suggest that the difficulty of handling the $l_{0}$ -norm does not only come from the nonconvexity but also from its gradient being zero or not well-defined. Accordingly, we propose desirable criteria to be a good nonconvex surrogate and suggest a corresponding family of surrogates. The proposed family of surrogates allows a simple regularizer, which enables efficient computation. The proposed surrogate embraces the benefits of both $l_{0}$ - and $l_{1}$ -norms, and most importantly, its gradient vanishes slowly, which allows stable optimization. We apply the proposed surrogate to well-known sparse representation problems and benchmark datasets to demonstrate its robustness and efficiency.

computer science, information systems,telecommunications,engineering, electrical & electronic
Gradient-based sparse principal component analysis with extensions to online learning

Yixuan Qiu,Jing Lei,Kathryn Roeder

DOI: https://doi.org/10.1093/biomet/asac041

IF: 3.0279

2022-07-12

Biometrika

Abstract:Summary Sparse principal component analysis is an important technique for simultaneous dimensionality reduction and variable selection with high-dimensional data. In this work we combine the unique geometric structure of the sparse principal component analysis problem with recent advances in convex optimization to develop novel gradient-based sparse principal component analysis algorithms. These algorithms enjoy the same global convergence guarantee as the original alternating direction method of multipliers, and can be more efficiently implemented with the rich toolbox developed for gradient methods from the deep learning literature. Most notably, these gradient-based algorithms can be combined with stochastic gradient descent methods to produce efficient online sparse principal component analysis algorithms with provable numerical and statistical performance guarantees. The practical performance and usefulness of the new algorithms are demonstrated in various simulation studies. As an application, we show how the scalability and statistical accuracy of our method enable us to find interesting functional gene groups in high-dimensional RNA sequencing data.

statistics & probability,mathematical & computational biology,biology
Learning No-Regret Sparse Generalized Linear Models with Varying Observation(s).

Diyang Li,Charles Ling,zhiqiang xu,Huan Xiong,Bin Gu

2024-01-01

Abstract:Generalized Linear Models (GLMs) encompass a wide array of regression and classification models, where prediction is a function of a linear combination of the input variables. Often in real-world scenarios, a number of observations would be added into or removed from the existing training dataset, necessitating the development of learning systems that can efficiently train optimal models with varying observations in an online (sequential) manner instead of retraining from scratch. Despite the significance of data-varying scenarios, most existing approaches to sparse GLMs concentrate on offline batch updates, leaving online solutions largely underexplored. In this work, we present the first algorithm without compromising accuracy for GLMs regularized by sparsity-enforcing penalties trained on varying observations. Our methodology is capable of handling the addition and deletion of observations simultaneously, while adaptively updating data-dependent regularization parameters to ensure the best statistical performance. Specifically, we recast sparse GLMs as a bilevel optimization objective upon varying observations and characterize it as an explicit gradient flow in the underlying space for the inner and outer subproblems we are optimizing over, respectively. We further derive a set of rules to ensure a proper transition at regions of non-smoothness, and establish the guarantees of theoretical consistency and finite convergence. Encouraging results are exhibited on real-world benchmarks.
OptG: Optimizing Gradient-driven Criteria in Network Sparsity

Yuxin Zhang,Mingbao Lin,Mengzhao Chen,Fei Chao,Rongrong Ji

DOI: https://doi.org/10.48550/arXiv.2201.12826

2022-01-30

Computer Vision and Pattern Recognition

Abstract:Network sparsity receives popularity mostly due to its capability to reduce the network complexity. Extensive studies excavate gradient-driven sparsity. Typically, these methods are constructed upon premise of weight independence, which however, is contrary to the fact that weights are mutually influenced. Thus, their performance remains to be improved. In this paper, we propose to optimize gradient-driven sparsity (OptG) by solving this independence paradox. Our motive comes from the recent advances in supermask training which shows that high-performing sparse subnetworks can be located by simply updating mask values without modifying any weight. We prove that supermask training is to accumulate the criteria of gradient-driven sparsity for both removed and preserved weights, and it can partly solve the independence paradox. Consequently, OptG integrates supermask training into gradient-driven sparsity, and a novel supermask optimizer is further proposed to comprehensively mitigate the independence paradox. Experiments show that OptG can well surpass many existing state-of-the-art competitors, especially at ultra-high sparsity levels. Our code is available at \url{https://github.com/zyxxmu/OptG}.
Gradient Sparsification For Masked Fine-Tuning of Transformers

James O' Neill,Sourav Dutta

2023-07-20

Abstract:Fine-tuning pretrained self-supervised language models is widely adopted for transfer learning to downstream tasks. Fine-tuning can be achieved by freezing gradients of the pretrained network and only updating gradients of a newly added classification layer, or by performing gradient updates on all parameters. Gradual unfreezing makes a trade-off between the two by gradually unfreezing gradients of whole layers during training. This has been an effective strategy to trade-off between storage and training speed with generalization performance. However, it is not clear whether gradually unfreezing layers throughout training is optimal, compared to sparse variants of gradual unfreezing which may improve fine-tuning performance. In this paper, we propose to stochastically mask gradients to regularize pretrained language models for improving overall fine-tuned performance. We introduce GradDrop and variants thereof, a class of gradient sparsification methods that mask gradients during the backward pass, acting as gradient noise. GradDrop is sparse and stochastic unlike gradual freezing. Extensive experiments on the multilingual XGLUE benchmark with XLMR-Large show that GradDrop is competitive against methods that use additional translated data for intermediate pretraining and outperforms standard fine-tuning and gradual unfreezing. A post-analysis shows how GradDrop improves performance with languages it was not trained on, such as under-resourced languages.

Computation and Language,Machine Learning

Sparse Online Learning via Truncated Gradient.

Quantum Algorithm for Sparse Online Learning with Truncated Gradient Descent

Online Kernel Learning with a Near Optimal Sparsity Bound

Screening for Sparse Online Learning

Learning with Non-Convex Truncated Losses by SGD

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

A Dual Perspective of Sparse and Robust Online Learning Algorithm

Gradient Sparsification for Communication-Efficient Distributed Optimization

Variable Screening for Sparse Online Regression.

Accelerating CNN Training by Sparsifying Activation Gradients

Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via $\ell_1, \ell_0$, and transformed-$\ell_1$ Penalties

Training a neural netwok for data reduction and better generalization

CONGO: Compressive Online Gradient Optimization

Noisy Truncated SGD: Optimization and Generalization

Batch gradient training method with smoothing ℓ_ 0 regularization for feedforward neural networks

Minimum Variance Unbiased N:M Sparsity for the Neural Gradients

Nonconvex Sparse Representation With Slowly Vanishing Gradient Regularizers

Gradient-based sparse principal component analysis with extensions to online learning

Learning No-Regret Sparse Generalized Linear Models with Varying Observation(s).

OptG: Optimizing Gradient-driven Criteria in Network Sparsity

Gradient Sparsification For Masked Fine-Tuning of Transformers