Abstract:The existing machine learning algorithms for minimizing the convex function over a closed convex set suffer from slow convergence because their learning rates must be determined before running them. This paper proposes two machine learning algorithms incorporating the line search method, which automatically and algorithmically finds appropriate learning rates at run-time. One algorithm is based on the incremental subgradient algorithm, which sequentially and cyclically uses each of the parts of the objective function; the other is based on the parallel subgradient algorithm, which uses parts independently in parallel. These algorithms can be applied to constrained nonsmooth convex optimization problems appearing in tasks of learning support vector machines without adjusting the learning rates precisely. The proposed line search method can determine learning rates to satisfy weaker conditions than the ones used in the existing machine learning algorithms. This implies that the two algorithms are generalizations of the existing incremental and parallel subgradient algorithms for solving constrained nonsmooth convex optimization problems. We show that they generate sequences that converge to a solution of the constrained nonsmooth convex optimization problem under certain conditions. The main contribution of this paper is the provision of three kinds of experiment showing that the two algorithms can solve concrete experimental problems faster than the existing algorithms. First, we show that the proposed algorithms have performance advantages over the existing ones in solving a test problem. Second, we compare the proposed algorithms with a different algorithm Pegasos, which is designed to learn with a support vector machine efficiently, in terms of prediction accuracy, value of the objective function, and computational time. Finally, we use one of our algorithms to train a multilayer neural network and discuss its applicability to deep learning.

Self-adaptive algorithms for quasiconvex programming and applications to machine learning

Adaptive multi-gradient methods for quasiconvex vector optimization and applications to multi-task learning

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Self-Adaptive Extragradient Algorithms for Quasi-Equilibrium Problems

Adaptive Subgradient Methods for Mathematical Programming Problems with Quasiconvex Functions

Adaptive Strategies in Non-convex Optimization

Self-adaptive inertial single projection methods for variational inequalities involving non-Lipschitz and Lipschitz operators with their applications to optimal control problems

A modified subgradient extragradient method with non-monotonic step sizes for solving quasimonotone variational inequalities

Accelerated Stochastic Optimization Methods under Quasar-convexity

On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithms

Modified Inertial Projection and Contraction Algorithms with Non-Monotonic Step Sizes for Solving Variational Inequalities and Their Applications

Beyond Convexity: Stochastic Quasi-Convex Optimization

An adaptive stochastic sequential quadratic programming with differentiable exact augmented lagrangians

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

Generalized Uniformly Optimal Methods for Nonlinear Programming

Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments

Extended convexity and smoothness and their applications in deep learning

Inertial self-adaptive algorithms for solving non-smooth convex optimization problems

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

A class of new three-term descent conjugate gradient algorithms for large-scale unconstrained optimization and applications to image restoration problems