Improving the communication in decentralized manifold optimization through single-step consensus and compression

Jiang Hu,Kangkang Deng

2024-07-12

Abstract:We are concerned with decentralized optimization over a compact submanifold, where the loss functions of local datasets are defined by their respective local datasets. A key challenge in decentralized optimization is mitigating the communication bottleneck, which primarily involves two strategies: achieving consensus and applying communication compression. Existing projection/retraction-type algorithms rely on multi-step consensus to attain both consensus and optimality. Due to the nonconvex nature of the manifold constraint, it remains an open question whether the requirement for multi-step consensus can be reduced to single-step consensus. We address this question by carefully elaborating on the smoothness structure and the asymptotic 1-Lipschitz continuity associated with the manifold constraint. Furthermore, we integrate these insights with a communication compression strategy to propose a communication-efficient gradient algorithm for decentralized manifold optimization problems, significantly reducing per-iteration communication costs. Additionally, we establish an iteration complexity of $\mathcal{O}(\epsilon^{-1})$ to find an $\epsilon$-stationary point, which matches the complexity in the Euclidean setting. Numerical experiments demonstrate the efficiency of the proposed method in comparison to state-of-the-art approaches.

Optimization and Control

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reduce the communication bottleneck in decentralized manifold optimization. Specifically, the paper focuses on the decentralized optimization problem on compact sub - manifolds, where the data set of each node defines its loss function. In decentralized optimization, one of the main challenges is to alleviate the communication bottleneck, which usually involves two strategies: reaching consensus and applying communication compression. Existing projection/retraction - type algorithms rely on multi - step consensus to achieve consensus and optimality. However, due to the non - convex nature of the manifold constraints, whether the requirement of multi - step consensus can be reduced to single - step consensus remains an open question. The paper addresses this issue by elaborating on the smooth structure and asymptotic 1 - Lipschitz continuity related to the manifold constraints. In addition, combined with the communication compression strategy, the paper also proposes a communication - efficient gradient algorithm for solving the decentralized manifold optimization problem, which significantly reduces the communication cost per iteration. The paper also establishes the iteration complexity \(O(\epsilon^{-1})\) of finding \(\epsilon\)-stable points, which matches the complexity in the Euclidean setting. Numerical experiments show that the proposed method is more efficient than existing methods.