Abstract:In Distributed optimization and Learning, and even more in the modern framework of federated learning, communication, which is slow and costly, is critical. We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats. LoCoDL works with a large class of unbiased compressors that includes widely-used sparsification and quantization methods. LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions. This is confirmed in practice, with LoCoDL outperforming existing algorithms.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the issue of communication efficiency in distributed optimization and learning, particularly within the framework of Federated Learning. Specifically, the paper proposes a new algorithm called LoCoDL, which combines Local Training (LT) and Communication Compression (CC) to improve communication efficiency in distributed learning. #### Main Issues: 1. **Communication Bottleneck**: In Federated Learning, communication between clients and the server is often slow and costly, which becomes the main bottleneck for large-scale deployment of Federated Learning. 2. **Limitations of Existing Methods**: Existing methods like FedAvg and Scaffold, although partially addressing the communication efficiency issue, still have shortcomings in terms of theoretical complexity and practical performance. #### Research Objectives: 1. **Combining the Advantages of LT and CC**: By simultaneously utilizing local training and communication compression techniques, the goal is to reduce communication frequency and compress communication information, thereby significantly lowering communication complexity. 2. **Theoretical Guarantees**: To prove that the LoCoDL algorithm has linear convergence in strongly convex settings and that its communication complexity is doubly accelerated, meaning that dependencies on both the condition number and model dimension are improved. 3. **Practical Performance Validation**: To experimentally validate the superior performance of the LoCoDL algorithm on different datasets and settings. In summary, LoCoDL aims to provide a communication-efficient distributed learning algorithm by combining local training and communication compression techniques, outperforming existing methods both theoretically and practically.

LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression

DiLoCo: Distributed Low-Communication Training of Language Models

LoCo: Low-Bit Communication Adaptor for Large-scale Model Training

Personalized Federated Learning with Communication Compression

Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression

Decentralized Deep Learning with Arbitrary Communication Compression

TAMUNA: Doubly Accelerated Distributed Optimization with Local Training, Compression, and Partial Participation

Communication-Efficient Distributed Learning with Local Immediate Error Compensation

Communication Compression for Distributed Learning without Control Variates

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models

Decentralized Federated Learning: Balancing Communication and Computing Costs

DoCoFL: Downlink Compression for Cross-Device Federated Learning

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Communication-Efficient Design for Quantized Decentralized Federated Learning

On the Convergence of Communication-Efficient Local SGD for Federated Learning

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

DualFL: A Duality-based Federated Learning Algorithm with Communication Acceleration in the General Convex Regime

Two-layer accumulated quantized compression for communication-efficient federated learning: TLAQC

FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning