Efficient Euclidean projections via Piecewise Root Finding and its application in gradient projection
Pinghua Gong,Kun Gai,Changshui Zhang
DOI: https://doi.org/10.1016/j.neucom.2011.02.019
IF: 6
2011-01-01
Neurocomputing
Abstract:Recently, the gradient (subgradient) projection method, especially by incorporating the idea of Nesterov's method, has aroused more and more attention and achieved great successes on constrained optimization problems arising in the field of machine learning, data mining and signal processing. In the gradient projection method, a critical step is how to efficiently project a vector onto a constraint set. In this paper, we propose a unified method called Piecewise Root Finding (PRF) to efficiently calculate Euclidean projections onto three typical constraint sets: @?"1-ball, Elastic Net (EN) and the Intersection of a Hyperplane and a Halfspace (IHH). In our PRF method, we first formulate a Euclidean projection problem as a root finding problem. Then, a Piecewise Root Finding algorithm is applied to find the root and global convergence is guaranteed. Finally, the Euclidean projection result is obtained as a function of the found root in a closed form. Moreover, the sparsity of the projected vector is considered, leading to reduced computational cost for projection onto the @?"1-ball and EN. Empirical studies demonstrate that our PRF algorithm is efficient by comparing it with several state of the art algorithms for Euclidean projections onto the three typical constraint sets mentioned above. Besides, we apply our efficient Euclidean projection algorithm (PRF) to the Gradient Projection with Nesterov's Method (GPNM), which efficiently solves the popular logistic regression problem with the @?"1-ball/EN/IHH constraint. Experimental results on real-world data sets indicate that GPNM has a fast convergence speed.