Abstract:Federated Edge Learning (FEL) is regarded as a promising distributed machine learning paradigm to reduce transmission latency and resources as well as preserve raw data privacy by collaboratively training local deep learning models across multiple edge devices. However, with the development of Artificial Intelligence (AI) technologies, the size of neural network models grows exponentially with their parameters to meet variable application requirements, which poses significant challenges to the computation, communication, and memory abilities of edge devices. Existing designs typically focus on either communication or computation efficiency without caring each device’s memory ability. To deal with the above issues, we first introduce a novel model sparse update enabled Tiny Federated Edge Learning (TinyFEL) architecture, which terminates the back-propagation early in local model training processes. Therefore, the proposed TinyFEL can reduce local memory occupation and lessen the communication-and-computation burden. Furthermore, we propose a parameter splitting mechanism instead of transmitting the full model, only a part of updated layers of parameters is transmitted for aggregation, which significantly reduced the communication overheads. Thereafter, we develop a communication and computation latency minimization problem to accelerate the training of TinyFEL. To this end, we theoretically analyze the convergence performance of TinyFEL, which unveils the mathematical relationship among sparse update ratio assignment, device selection, and learning performance. Then, a joint sparse update ratio assignment, device selection, and resource allocation strategy is introduced based on the Alternating Direction Method of Multipliers (ADMM) and Block Coordinate Descent (BCD) algorithms. Numerical results indicate that our proposed TinyFEL can reduce training memory occupation by over 40% than the traditional FEL at the cost of negligible accuracy loss.

TinyFL: A Lightweight Federated Learning Method with Efficient Memory-and-Communication.

Message Passing Based Wireless Federated Learning Via Analog Message Aggregation

TinyFEL: Communication, Computation, and Memory Efficient Tiny Federated Edge Learning Via Model Sparse Update

Asynchronous Federated Learning over Wireless Communication Networks.

Federated Learning over Wireless IoT Networks with Optimized Communication and Resources

Communication-Efficient Federated Learning in Channel Constrained Internet of Things

Split-FL: An Efficient Online Federated Learning Framework with Constrained Computation and Streaming Data

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Snowball: Energy Efficient and Accurate Federated Learning with Coarse-to-Fine Compression over Heterogeneous Wireless Edge Devices

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Efficient and Less Centralized Federated Learning

Over-the-Air Federated Learning and Optimization

Efficient Wireless Federated Learning with Partial Model Aggregation

Communication and Energy Efficient Slimmable Federated Learning via Superposition Coding and Successive Decoding

Adaptive Model Pruning and Personalization for Federated Learning Over Wireless Networks

To talk or to work: Flexible communication compression for energy efficient federated learning over heterogeneous mobile edge devices

IoT Device Friendly and Communication Efficient Federated Learning via Joint Model Pruning and Quantization

NeuLite: Memory-Efficient Federated Learning via Elastic Progressive Training

Time-triggered Federated Learning over Wireless Networks

FedPE: Adaptive Model Pruning-Expanding for Federated Learning on Mobile Devices

Efficient Adaptive Federated Learning in Resource-Constrained IoT Environments