Abstract:Distributed second-order optimization, as an effective strategy for training large-scale machine learning systems, has been widely investigated due to its low communication complexity. However, the existing distributed second-order optimization algorithms, including distributed approximate Newton (DANE), accelerated inexact DANE (AIDE), and statistically preconditioned accelerated gradient (SPAG), are all required to precisely solve an expensive subproblem up to the target precision. Therefore, this causes these algorithms to suffer from high computation costs and this hinders their development. In this article, we design a novel distributed second-order algorithm called the accelerated distributed approximate Newton (ADAN) method to overcome the high computation costs of the existing ones. Compared with DANE, AIDE, and SPAG, which are constructed based on the relative smooth theory, ADAN's theoretical foundation is built upon the inexact Newton theory. The different theoretical foundations lead to handle the expensive subproblem efficiently, and steps required to solve the subproblem are independent of the target precision. At the same time, ADAN resorts to the acceleration and can effectively exploit the objective function's curvature information, making ADAN to achieve a low communication complexity. Thus, ADAN can achieve both the communication and computation efficiencies, while DANE, AIDE, and SPAG can achieve only the communication efficiency. Our empirical study also validates the advantages of ADAN over extant distributed second-order algorithms.

Adaptive pruning-based Newton's method for distributed learning

Adaptive Newton's Method for Distributed Optimization

Resource-Adaptive Newton's Method for Distributed Learning

Distributed Adaptive Newton Methods with Globally Superlinear Convergence

Distributed adaptive Newton methods with global superlinear convergence

Distributed Newton Methods for Deep Neural Networks

Achieving Globally Superlinear Convergence for Distributed Optimization with Adaptive Newton Method

Accelerated Distributed Approximate Newton Method

Quasi-Newton Updating for Large-Scale Distributed Learning

Distributed Inexact Newton Method with Adaptive Step Sizes

On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyond

Distributed Learning with Convex SUM-of -Non-convex Objective

On the Convergence of Decentralized Adaptive Gradient Methods

Online Learning over Distributed Low-Rank Networks Via Sequential Power Iteration

Distributed Inexact Newton-Type Pursuit For Non-Convex Sparse Learning

Distributed Adaptive Greedy Quasi-Newton Methods with Explicit Non-asymptotic Convergence Bounds

Network Newton-Part II: Convergence Rate and Implementation

Distributed Adaptive Subgradient Algorithms for Online Learning over Time-Varying Networks

Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

Distributed Newton Methods for Strictly Convex Consensus Optimization Problems in Multi-Agent Networks