On the Universality of Online Mirror Descent

Nathan Srebro,Karthik Sridharan,Ambuj Tewari

DOI: https://doi.org/10.48550/arXiv.1107.4080

2011-07-21

Abstract:We show that for a general class of convex online learning problems, Mirror Descent can always achieve a (nearly) optimal regret guarantee.

Machine Learning

What problem does this paper attempt to address?

Non-convex online learning via algorithmic equivalence

Udaya Ghai,Zhou Lu,Elad Hazan

DOI: https://doi.org/10.48550/arXiv.2205.15235

IF: 5.414

2022-05-30

Machine Learning

Abstract:We study an algorithmic equivalence technique between non-convex gradient descent and convex mirror descent. We start by looking at a harder problem of regret minimization in online non-convex optimization. We show that under certain geometric and smoothness conditions, online gradient descent applied to non-convex functions is an approximation of online mirror descent applied to convex functions under reparameterization. In continuous time, the gradient flow with this reparameterization was shown to be exactly equivalent to continuous-time mirror descent by Amid and Warmuth 2020, but theory for the analogous discrete time algorithms is left as an open problem. We prove an $O(T^{\frac{2}{3}})$ regret bound for non-convex online gradient descent in this setting, answering this open problem. Our analysis is based on a new and simple algorithmic equivalence method.
The Information Geometry of Mirror Descent

Garvesh Raskutti,Sayan Mukherjee

DOI: https://doi.org/10.1109/tit.2015.2388583

IF: 2.5

2015-03-01

IEEE Transactions on Information Theory

Abstract:We prove the equivalence of two online learning algorithms: 1) mirror descent and 2) natural gradient descent. Both mirror descent and natural gradient descent are generalizations of online gradient descent when the parameter of interest lies on a non-Euclidean manifold. Natural gradient descent selects the steepest descent along a Riemannian manifold by multiplying the standard gradient by the inverse of the metric tensor. Mirror descent induces non-Euclidean structure by solving iterative optimization problems using different proximity functions. In this paper, we prove that mirror descent induced by Bregman divergence proximity functions is equivalent to the natural gradient descent algorithm on the dual Riemannian manifold. We use techniques from convex analysis and connections between Riemannian manifolds, Bregman divergences, and convexity to prove this result. This equivalence between natural gradient descent and mirror descent, implies that: 1) mirror descent is the steepest descent direction along the Riemannian manifold corresponding to the choice of Bregman divergence and 2) mirror descent with log-likelihood loss applied to parameter estimation in exponential families asymptotically achieves the classical Cramér–Rao lower bound.

computer science, information systems,engineering, electrical & electronic
Parameter-free Mirror Descent

Andrew Jacobsen,Ashok Cutkosky

2024-02-09

Abstract:We develop a modified online mirror descent framework that is suitable for building adaptive and parameter-free algorithms in unbounded domains. We leverage this technique to develop the first unconstrained online linear optimization algorithm achieving an optimal dynamic regret bound, and we further demonstrate that natural strategies based on Follow-the-Regularized-Leader are unable to achieve similar results. We also apply our mirror descent framework to build new parameter-free implicit updates, as well as a simplified and improved unconstrained scale-free algorithm.

Machine Learning,Optimization and Control
Improving Dynamic Regret in Distributed Online Mirror Descent Using Primal and Dual Information

Nima Eshraghi,Ben Liang

DOI: https://doi.org/10.48550/arXiv.2112.03504

2021-12-07

Abstract:We consider the problem of distributed online optimization, with a group of learners connected via a dynamic communication graph. The goal of the learners is to track the global minimizer of a sum of time-varying loss functions in a distributed manner. We propose a novel algorithm, termed Distributed Online Mirror Descent with Multiple Averaging Decision and Gradient Consensus (DOMD-MADGC), which is based on mirror descent but incorporates multiple consensus averaging iterations over local gradients as well as local decisions. The key idea is to allow the local learners to collect a sufficient amount of global information, which enables them to more accurately approximation the time-varying global loss, so that they can closely track the dynamic global minimizer over time. We show that the dynamic regret of DOMD-MADGC is upper bounded by the path length, which is defined as the cumulative distance between successive minimizers. The resulting bound improves upon the bounds of existing distributed online algorithms and removes the explicit dependence on $T$.

Optimization and Control,Distributed, Parallel, and Cluster Computing
Mirror Descent Meets Fixed Share (and feels no regret)

Nicolò Cesa-Bianchi,Pierre Gaillard,Gabor Lugosi,Gilles Stoltz

DOI: https://doi.org/10.48550/arXiv.1202.3323

IF: 5.414

2012-02-15

Machine Learning

Abstract:Mirror descent with an entropic regularizer is known to achieve shifting regret bounds that are logarithmic in the dimension. This is done using either a carefully designed projection or by a weight sharing technique. Via a novel unified analysis, we show that these two approaches deliver essentially equivalent bounds on a notion of regret generalizing shifting, adaptive, discounted, and other related regrets. Our analysis also captures and extends the generalized weight sharing technique of Bousquet and Warmuth, and can be refined in several ways, including improvements for small losses and adaptive tuning of parameters.
Inexact Online Proximal Mirror Descent for time-varying composite optimization

Woocheol Choi,Myeong-Su Lee,Seok-Bae Yun

2023-04-11

Abstract:In this paper, we consider the online proximal mirror descent for solving the time-varying composite optimization problems. For various applications, the algorithm naturally involves the errors in the gradient and proximal operator. We obtain sharp estimates on the dynamic regret of the algorithm when the regular part of the cost is convex and smooth. If the Bregman distance is given by the Euclidean distance, our result also improves the previous work in two ways: (i) We establish a sharper regret bound compared to the previous work in the sense that our estimate does not involve $O(T)$ term appearing in that work. (ii) We also obtain the result when the domain is the whole space $\mathbb{R}^n$, whereas the previous work was obtained only for bounded domains. We also provide numerical tests for problems involving the errors in the gradient and proximal operator.

Optimization and Control
Connections Between Mirror Descent, Thompson Sampling and the Information Ratio

Julian Zimmert,Tor Lattimore

DOI: https://doi.org/10.48550/arXiv.1905.11817

2019-05-28

Abstract:The information-theoretic analysis by Russo and Van Roy (2014) in combination with minimax duality has proved a powerful tool for the analysis of online learning algorithms in full and partial information settings. In most applications there is a tantalising similarity to the classical analysis based on mirror descent. We make a formal connection, showing that the information-theoretic bounds in most applications can be derived from existing techniques for online convex optimisation. Besides this, for $k$-armed adversarial bandits we provide an efficient algorithm with regret that matches the best information-theoretic upper bound and improve best known regret guarantees for online linear optimisation on $\ell_p$-balls and bandits with graph feedback.

Machine Learning
Online Self-Concordant and Relatively Smooth Minimization, With Applications to Online Portfolio Selection and Learning Quantum States

Chung-En Tsai,Hao-Chung Cheng,Yen-Huan Li

2023-09-21

Abstract:Consider an online convex optimization problem where the loss functions are self-concordant barriers, smooth relative to a convex function $h$, and possibly non-Lipschitz. We analyze the regret of online mirror descent with $h$. Then, based on the result, we prove the following in a unified manner. Denote by $T$ the time horizon and $d$ the parameter dimension. 1. For online portfolio selection, the regret of $\widetilde{\text{EG}}$, a variant of exponentiated gradient due to Helmbold et al., is $\tilde{O} ( T^{2/3} d^{1/3} )$ when $T > 4 d / \log d$. This improves on the original $\tilde{O} ( T^{3/4} d^{1/2} )$ regret bound for $\widetilde{\text{EG}}$. 2. For online portfolio selection, the regret of online mirror descent with the logarithmic barrier is $\tilde{O}(\sqrt{T d})$. The regret bound is the same as that of Soft-Bayes due to Orseau et al. up to logarithmic terms. 3. For online learning quantum states with the logarithmic loss, the regret of online mirror descent with the log-determinant function is also $\tilde{O} ( \sqrt{T d} )$. Its per-iteration time is shorter than all existing algorithms we know.

Machine Learning,Optimization and Control,Portfolio Management,Quantum Physics
A Unified View of Regularized Dual Averaging and Mirror Descent with Implicit Updates

H. Brendan McMahan

DOI: https://doi.org/10.48550/arXiv.1009.3240

2011-09-21

Abstract:We study three families of online convex optimization algorithms: follow-the-proximally-regularized-leader (FTRL-Proximal), regularized dual averaging (RDA), and composite-objective mirror descent. We first prove equivalence theorems that show all of these algorithms are instantiations of a general FTRL update. This provides theoretical insight on previous experimental observations. In particular, even though the FOBOS composite mirror descent algorithm handles L1 regularization explicitly, it has been observed that RDA is even more effective at producing sparsity. Our results demonstrate that FOBOS uses subgradient approximations to the L1 penalty from previous rounds, leading to less sparsity than RDA, which handles the cumulative penalty in closed form. The FTRL-Proximal algorithm can be seen as a hybrid of these two, and outperforms both on a large, real-world dataset. Our second contribution is a unified analysis which produces regret bounds that match (up to logarithmic terms) or improve the best previously known bounds. This analysis also extends these algorithms in two important ways: we support a more general type of composite objective and we analyze implicit updates, which replace the subgradient approximation of the current loss function with an exact optimization.

Machine Learning
Sparse Q-learning with Mirror Descent

Sridhar Mahadevan,Bo Liu

DOI: https://doi.org/10.48550/arXiv.1210.4893

IF: 5.414

2012-10-16

Machine Learning

Abstract:This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional gradient methods, mirror descent undertakes gradient updates of weights in both the dual space and primal space, which are linked together using a Legendre transform. Mirror descent can be viewed as a proximal algorithm where the distance generating function used is a Bregman divergence. A new class of proximal-gradient based temporal-difference (TD) methods are presented based on different Bregman divergences, which are more powerful than regular TD learning. Examples of Bregman divergences that are studied include p-norm functions, and Mahalanobis distance based on the covariance of sample gradients. A new family of sparse mirror-descent reinforcement learning methods are proposed, which are able to find sparse fixed points of an l1-regularized Bellman equation at significantly less computational cost than previous methods based on second-order matrix methods. An experimental study of mirror-descent reinforcement learning is presented using discrete and continuous Markov decision processes.
Gradient-Variation Online Learning under Generalized Smoothness

Yan-Feng Xie,Peng Zhao,Zhi-Hua Zhou

2024-11-03

Abstract:Gradient-variation online learning aims to achieve regret guarantees that scale with variations in the gradients of online functions, which has been shown to be crucial for attaining fast convergence in games and robustness in stochastic optimization, hence receiving increased attention. Existing results often require the smoothness condition by imposing a fixed bound on gradient Lipschitzness, which may be unrealistic in practice. Recent efforts in neural network optimization suggest a generalized smoothness condition, allowing smoothness to correlate with gradient norms. In this paper, we systematically study gradient-variation online learning under generalized smoothness. We extend the classic optimistic mirror descent algorithm to derive gradient-variation regret by analyzing stability over the optimization trajectory and exploiting smoothness locally. Then, we explore universal online learning, designing a single algorithm with the optimal gradient-variation regrets for convex and strongly convex functions simultaneously, without requiring prior knowledge of curvature. This algorithm adopts a two-layer structure with a meta-algorithm running over a group of base-learners. To ensure favorable guarantees, we design a new Lipschitz-adaptive meta-algorithm, capable of handling potentially unbounded gradients while ensuring a second-order bound to effectively ensemble the base-learners. Finally, we provide the applications for fast-rate convergence in games and stochastic extended adversarial optimization.

Machine Learning,Optimization and Control
Banker Online Mirror Descent

Jiatai Huang,Longbo Huang

DOI: https://doi.org/10.48550/arxiv.2106.08943

2023-01-01

Abstract: We propose Banker-OMD, a novel framework generalizing the classical Online Mirror Descent (OMD) technique in online learning algorithm design. Banker-OMD allows algorithms to robustly handle delayed feedback, and offers a general methodology for achieving $\tilde{O}(\sqrt{T} + \sqrt{D})$-style regret bounds in various delayed-feedback online learning tasks, where $T$ is the time horizon length and $D$ is the total feedback delay. We demonstrate the power of Banker-OMD with applications to three important bandit scenarios with delayed feedback, including delayed adversarial Multi-armed bandits (MAB), delayed adversarial linear bandits, and a novel delayed best-of-both-worlds MAB setting. Banker-OMD achieves nearly-optimal performance in all the three settings. In particular, it leads to the first delayed adversarial linear bandit algorithm achieving $\tilde{O}(\text{poly}(n)(\sqrt{T} + \sqrt{D}))$ regret.
Universal Online Convex Optimization with Minimax Optimal Second-Order Dynamic Regret

Hakan Gokcesu,Suleyman S. Kozat

DOI: https://doi.org/10.1109/tac.2024.3381866

IF: 6.549

2024-01-01

IEEE Transactions on Automatic Control

Abstract:We introduce an online convex optimization algorithm which utilizes projected subgradient descent with optimal adaptive learning rates. Our method provides second-order minimax-optimal dynamic regret guarantee (i.e., dependent on the sum of squared subgradient norms) for a sequence of general convex functions, which may not have strong-convexity, smoothness, exp-concavity or even proper Lipschitz-continuity. The regret guarantee is against any comparator decision sequence with bounded path variation (i.e., sum of the distances between successive decisions). We generate the lower bound of the worst-case second-order dynamic regret by incorporating actual subgradient norms. We show that this lower bound matches with our regret guarantee within a constant factor, which makes our algorithm minimax optimal. We also derive the extension for learning in each decision coordinate individually. We demonstrate how to best preserve our regret guarantee in a truly online manner, when the bound on path variation of the comparator sequence grows in time or the feedback regarding such bound arrives partially as time goes on. We further build on our algorithm to eliminate the need of any knowledge on the comparator path variation, and provide minimax optimal second-order regret guarantees with no a priori information. Our approach can compete against all comparator sequences simultaneously (universally) in a minimax optimal manner, i.e., each regret guarantee depends on the respective comparator path variation. We discuss modifications to our approach which address complexity reductions for time, computation and memory. We further improve our results by making the regret guarantees also dependent on comparator sets' diameters in addition to the respective path variations.

automation & control systems,engineering, electrical & electronic
Multi-class classification: mirror descent approach

Daria Reshetova

DOI: https://doi.org/10.48550/arXiv.1607.00076

2016-12-08

Abstract:We consider the problem of multi-class classification and a stochastic opti- mization approach to it. We derive risk bounds for stochastic mirror descent algorithm and provide examples of set geometries that make the use of the algorithm efficient in terms of error in k.

Optimization and Control,Machine Learning
Investigating Variance Definitions for Mirror Descent with Relative Smoothness

Hadrien Hendrikx

2024-04-18

Abstract:Mirror Descent is a popular algorithm, that extends Gradients Descent (GD) beyond the Euclidean geometry. One of its benefits is to enable strong convergence guarantees through smooth-like analyses, even for objectives with exploding or vanishing curvature. This is achieved through the introduction of the notion of relative smoothness, which holds in many of the common use-cases of Mirror descent. While basic deterministic results extend well to the relative setting, most existing stochastic analyses require additional assumptions on the mirror, such as strong convexity (in the usual sense), to ensure bounded variance. In this work, we revisit Stochastic Mirror Descent (SMD) proofs in the (relatively-strongly-) convex and relatively-smooth setting, and introduce a new (less restrictive) definition of variance which can generally be bounded (globally) under mild regularity assumptions. We then investigate this notion in more details, and show that it naturally leads to strong convergence guarantees for stochastic mirror descent. Finally, we leverage this new analysis to obtain convergence guarantees for the Maximum Likelihood Estimator of a Gaussian with unknown mean and variance.

Optimization and Control
A Unified Approach to Controlling Implicit Regularization via Mirror Descent

Haoyuan Sun,Khashayar Gatmiry,Kwangjun Ahn,Navid Azizan

2024-01-11

Abstract:Inspired by the remarkable success of large neural networks, there has been significant interest in understanding the generalization performance of over-parameterized models. Substantial efforts have been invested in characterizing how optimization algorithms impact generalization through their "preferred" solutions, a phenomenon commonly referred to as implicit regularization. In particular, it has been argued that gradient descent (GD) induces an implicit $\ell_2$-norm regularization in regression and classification problems. However, the implicit regularization of different algorithms are confined to either a specific geometry or a particular class of learning problems, indicating a gap in a general approach for controlling the implicit regularization. To address this, we present a unified approach using mirror descent (MD), a notable generalization of GD, to control implicit regularization in both regression and classification settings. More specifically, we show that MD with the general class of homogeneous potential functions converges in direction to a generalized maximum-margin solution for linear classification problems, thereby answering a long-standing question in the classification setting. Further, we show that MD can be implemented efficiently and enjoys fast convergence under suitable conditions. Through comprehensive experiments, we demonstrate that MD is a versatile method to produce learned models with different regularizers, which in turn have different generalization performances.

Machine Learning
Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach.

Yu-Hu Yan,Peng Zhao,Zhi-Hua Zhou

2023-01-01

Abstract:In this paper, we propose an online convex optimization approach with two different levels of adaptivity. On a higher level, our approach is agnostic to the unknown types and curvatures of the online functions, while at a lower level, it can exploit the unknown niceness of the environments and attain problem-dependent guarantees. Specifically, we obtain $\mathcal{O}(\log V_T)$, $\mathcal{O}(d \log V_T)$ and $\hat{\mathcal{O}}(\sqrt{V_T})$ regret bounds for strongly convex, exp-concave and convex loss functions, respectively, where $d$ is the dimension, $V_T$ denotes problem-dependent gradient variations and the $\hat{\mathcal{O}}(\cdot)$-notation omits $\log V_T$ factors. Our result not only safeguards the worst-case guarantees but also directly implies the small-loss bounds in analysis. Moreover, when applied to adversarial/stochastic convex optimization and game theory problems, our result enhances the existing universal guarantees. Our approach is based on a multi-layer online ensemble framework incorporating novel ingredients, including a carefully designed optimism for unifying diverse function types and cascaded corrections for algorithmic stability. Notably, despite its multi-layer structure, our algorithm necessitates only one gradient query per round, making it favorable when the gradient evaluation is time-consuming. This is facilitated by a novel regret decomposition equipped with carefully designed surrogate losses.
Online distributed optimization with stochastic gradients: high probability bound of regrets

Yuchen Yang,Kaihong Lu,Long Wang

DOI: https://doi.org/10.1007/s11768-023-00186-3

2024-01-05

Control Theory and Technology

Abstract:In this paper, the problem of online distributed optimization subject to a convex set is studied via a network of agents. Each agent only has access to a noisy gradient of its own objective function, and can communicate with its neighbors via a network. To handle this problem, an online distributed stochastic mirror descent algorithm is proposed. Existing works on online distributed algorithms involving stochastic gradients only provide the expectation bounds of the regrets. Different from them, we study the high probability bound of the regrets, i.e., the sublinear bound of the regret is characterized by the natural logarithm of the failure probability's inverse. Under mild assumptions on the graph connectivity, we prove that the dynamic regret grows sublinearly with a high probability if the deviation in the minimizer sequence is sublinear with the square root of the time horizon. Finally, a simulation is provided to demonstrate the effectiveness of our theoretical results.

automation & control systems
Banker Online Mirror Descent: A Universal Approach for Delayed Online Bandit Learning

Jiatai Huang,Yan Dai,Longbo Huang

DOI: https://doi.org/10.48550/arxiv.2301.10500

2023-01-01

Abstract:We propose Banker Online Mirror Descent (Banker-OMD), a novel framework generalizing the classical Online Mirror Descent (OMD) technique in the online learning literature. The Banker-OMD framework almost completely decouples feedback delay handling and the task-specific OMD algorithm design, thus facilitating the design of new algorithms capable of efficiently and robustly handling feedback delays. Specifically, it offers a general methodology for achieving $\widetilde{\mathcal O}(\sqrt{T} + \sqrt{D})$-style regret bounds in online bandit learning tasks with delayed feedback, where $T$ is the number of rounds and $D$ is the total feedback delay. We demonstrate the power of Banker-OMD by applications to two important bandit learning scenarios with delayed feedback, including delayed scale-free adversarial Multi-Armed Bandits (MAB) and delayed adversarial linear bandits. Banker-OMD leads to the first delayed scale-free adversarial MAB algorithm achieving $\widetilde{\mathcal O}(\sqrt{K}L(\sqrt T+\sqrt D))$ regret and the first delayed adversarial linear bandit algorithm achieving $\widetilde{\mathcal O}(\text{poly}(n)(\sqrt{T} + \sqrt{D}))$ regret. As a corollary, the first application also implies $\widetilde{\mathcal O}(\sqrt{KT}L)$ regret for non-delayed scale-free adversarial MABs, which is the first to match the $\Omega(\sqrt{KT}L)$ lower bound up to logarithmic factors and can be of independent interest.
Optimistic Online Mirror Descent for Bridging Stochastic and Adversarial Online Convex Optimization

Sijia Chen,Yu-Jie Zhang,Wei-Wei Tu,Peng Zhao,Lijun Zhang

2024-03-16

Abstract:Stochastically Extended Adversarial (SEA) model is introduced by Sachs et al. [2022] as an interpolation between stochastic and adversarial online convex optimization. Under the smoothness condition, they demonstrate that the expected regret of optimistic follow-the-regularized-leader (FTRL) depends on the cumulative stochastic variance $\sigma_{1:T}^2$ and the cumulative adversarial variation $\Sigma_{1:T}^2$ for convex functions. They also provide a slightly weaker bound based on the maximal stochastic variance $\sigma_{\max}^2$ and the maximal adversarial variation $\Sigma_{\max}^2$ for strongly convex functions. Inspired by their work, we investigate the theoretical guarantees of optimistic online mirror descent (OMD) for the SEA model. For convex and smooth functions, we obtain the same $\mathcal{O}(\sqrt{\sigma_{1:T}^2}+\sqrt{\Sigma_{1:T}^2})$ regret bound, without the convexity requirement of individual functions. For strongly convex and smooth functions, we establish an $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2) \log (\sigma_{1:T}^2+\Sigma_{1:T}^2))$ bound, better than their $\mathcal{O}((\sigma_{\max}^2 + \Sigma_{\max}^2) \log T)$ result. For exp-concave and smooth functions, we achieve a new $\mathcal{O}(d\log(\sigma_{1:T}^2+\Sigma_{1:T}^2))$ bound. Owing to the OMD framework, we broaden our work to study dynamic regret minimization and scenarios where the online functions are non-smooth. We establish the first dynamic regret guarantee for the SEA model with convex and smooth functions, which is more favorable than static regret bounds in non-stationary scenarios. Furthermore, to deal with non-smooth and convex functions in the SEA model, we propose novel algorithms building on optimistic OMD with an implicit update, which provably attain static regret and dynamic regret guarantees without smoothness conditions.

Machine Learning

On the Universality of Online Mirror Descent

Non-convex online learning via algorithmic equivalence

The Information Geometry of Mirror Descent

Parameter-free Mirror Descent

Improving Dynamic Regret in Distributed Online Mirror Descent Using Primal and Dual Information

Mirror Descent Meets Fixed Share (and feels no regret)

Inexact Online Proximal Mirror Descent for time-varying composite optimization

Connections Between Mirror Descent, Thompson Sampling and the Information Ratio

Online Self-Concordant and Relatively Smooth Minimization, With Applications to Online Portfolio Selection and Learning Quantum States

A Unified View of Regularized Dual Averaging and Mirror Descent with Implicit Updates

Sparse Q-learning with Mirror Descent

Gradient-Variation Online Learning under Generalized Smoothness

Banker Online Mirror Descent

Universal Online Convex Optimization with Minimax Optimal Second-Order Dynamic Regret

Multi-class classification: mirror descent approach

Investigating Variance Definitions for Mirror Descent with Relative Smoothness

A Unified Approach to Controlling Implicit Regularization via Mirror Descent

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach.

Online distributed optimization with stochastic gradients: high probability bound of regrets

Banker Online Mirror Descent: A Universal Approach for Delayed Online Bandit Learning

Optimistic Online Mirror Descent for Bridging Stochastic and Adversarial Online Convex Optimization