Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1679-1707, June 2024. Classical primal-dual algorithms attempt to solve [math] by alternately minimizing over the primal variable [math] through primal descent and maximizing the dual variable [math] through dual ascent. However, when [math] is highly nonconvex with complex constraints in [math], the minimization over [math] may not achieve global optimality and, hence, the dual ascent step loses its valid intuition. This observation motivates us to propose a new class of primal-dual algorithms for nonconvex constrained optimization with the key feature to reverse dual ascent to a conceptually new dual descent, in a sense, elevating the dual variable to the same status as the primal variable. Surprisingly, this new dual scheme achieves some best iteration complexities for solving nonconvex optimization problems. In particular, when the dual descent step is scaled by a fractional constant, we name it scaled dual descent (SDD), otherwise, unscaled dual descent (UDD). For nonconvex multiblock optimization with nonlinear equality constraints, we propose SDD-alternating direction method of multipliers (SDD-ADMM) and show that it finds an [math]-stationary solution in [math] iterations. The complexity is further improved to [math] and [math] under proper conditions. We also propose UDD-augmented Lagrangian method (UDD-ALM), combining UDD with ALM, for weakly convex minimization over affine constraints. We show that UDD-ALM finds an [math]-stationary solution in [math] iterations. These complexity bounds for both algorithms either achieve or improve the best-known results in the ADMM and ALM literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing ADMM frameworks.

Dual Active Set Method for Support Vector Machines under Multi-Constraint Activation

Recursive Training Algorithm for One-Class Support Vector Machine Based on Active Set Method

Secondary Descent Active Set Algorithm Based on SVM

A Multi-Class Large Margin Classifier

Multitask Multiclass Support Vector Machines

Two New Decomposition Algorithms for Training Bound-Constrained Support Vector Machines

An accelerated active-set algorithm for a quadratic semidefinite program with general constraints

New method for solving Ivanov regularization-based support vector machine learning

A Simplified Multi-Class Support Vector Machine with Reduced Dual Optimization

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines

A Confident Majority Voting Strategy for Parallel and Modular Support Vector Machines

Improved Conjugate Gradient Implementation for Least Squares Support Vector Machines

A Fast Iterative Single Data Approach to Training Unconstrained Least Squares Support Vector Machines

A Subspace Minimization Barzilai-Borwein Method for Multiobjective Optimization Problems

Nonlinear Combination of Multiple Kernels for Support Vector Machines

A Minimization Approach for Minimax Optimization with Coupled Constraints

A New Active Set Scheme for Quadratic Programing

An efficient active-set method with applications to sparse approximations and risk minimization

Fast Algorithms for Linear and Kernel SVM+.