Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation

Rustem Islamov,Xun Qian,Slavomír Hanzely,Mher Safaryan,Peter Richtárik

DOI: https://doi.org/10.48550/arXiv.2206.03588

2022-06-08

Abstract:Despite their high computation and communication costs, Newton-type methods remain an appealing option for distributed training due to their robustness against ill-conditioned convex problems. In this work, we study ommunication compression and aggregation mechanisms for curvature information in order to reduce these costs while preserving theoretically superior local convergence guarantees. We prove that the recently developed class of three point compressors (3PC) of Richtarik et al. [2022] for gradient communication can be generalized to Hessian communication as well. This result opens up a wide variety of communication strategies, such as contractive compression} and lazy aggregation, available to our disposal to compress prohibitively costly curvature information. Moreover, we discovered several new 3PC mechanisms, such as adaptive thresholding and Bernoulli aggregation, which require reduced communication and occasional Hessian computations. Furthermore, we extend and analyze our approach to bidirectional communication compression and partial device participation setups to cater to the practical considerations of applications in federated learning. For all our methods, we derive fast condition-number-independent local linear and/or superlinear convergence rates. Finally, with extensive numerical evaluations on convex optimization problems, we illustrate that our designed schemes achieve state-of-the-art communication complexity compared to several key baselines using second-order information.

Machine Learning,Optimization and Control

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the high computational and communication costs of Newton - type methods in distributed training. Specifically, the author aims to reduce these costs by introducing communication compression and aggregation mechanisms while maintaining theoretically superior local convergence guarantees. The following is a specific explanation of this problem:

### Research Background
In distributed optimization, Newton - type methods are attractive because of their robustness to ill - conditioned convex problems. However, these methods are usually accompanied by high computational and communication costs. Especially in cases involving the Hessian matrix (second - order derivative), the communication cost is particularly significant.

### Paper Objectives
To meet this challenge, the paper studies communication compression and aggregation mechanisms for curvature information (such as the Hessian matrix) to reduce communication costs while ensuring theoretically fast local convergence. Specifically, the author hopes:
1. **Reduce communication overhead**: By compressing the communication volume of the Hessian matrix, reduce the communication burden in the distributed system.
2. **Maintain convergence speed**: Ensure that the proposed method can theoretically achieve a fast local linear and super - linear convergence rate similar to the classical Newton method.
3. **Adapt to practical application requirements**: Considering the practical requirements of application scenarios such as federated learning, extend the method to support two - way communication compression and partial device participation settings.

### Main Contributions
1. **Generalize the Three - Point Compressor (3PC)**: Prove that the Three - Point Compressor (3PC) in gradient communication proposed by Richtárik et al. can be generalized to Hessian communication. This makes it possible to use various compression strategies such as shrinkage compression and lazy aggregation.
2. **New compression and aggregation schemes**: Propose several new 3PC mechanisms, such as adaptive thresholding and Bernoulli aggregation, which can reduce the communication volume and occasionally perform Hessian calculations.
3. **Extension to two - way communication compression**: Extend the method to two - way communication compression and partial device participation settings to meet the practical requirements of federated learning.
4. **Fast local convergence rate**: Under the assumptions of strong convexity and local Hessian Lipschitz continuity, derive fast condition - number - independent local linear and/or super - linear convergence rates.
5. **Experimental verification**: Through extensive numerical experiments, show that the designed scheme achieves the state - of - the - art communication complexity on convex optimization problems and performs well compared to other baseline methods using second - order information.

### Formula Representation
The formulas involved in the paper are represented in Markdown format as follows:

- Definition of the Three - Point Compressor for the Hessian matrix:
\[
E\left[\|C_{H,Y}(X) - X\|_F^2\right] \leq (1 - A)\|H - Y\|_F^2 + B\|X - Y\|_F^2
\]
where \(0 < A \leq 1\) and \(B \geq 0\).

- Definition of adaptive thresholding:
\[
[C(X)]_{jl} =
\begin{cases}
X_{jl} & \text{if } |X_{jl}| \geq \lambda \|X\|_\infty \\
0 & \text{otherwise}
\end{cases}
\]

- Definition of the shrinkage compressor:
\[
E\left[\|C(X) - X\|_F^2\right] \leq (1 - \alpha)\|X\|_F^2
\]

Through these improvements, the paper effectively solves the high communication and computational cost problems in distributed Newton - type methods and provides theoretical and experimental support for practical applications.