Abstract:Low-rank model compression is a widely used technique for reducing the computational load when training machine learning models. However, existing methods often rely on relaxing the low-rank constraint of the model weights using a regularized nuclear norm penalty, which requires an appropriate hyperparameter that can be difficult to determine in practice. Furthermore, existing compression techniques are not directly applicable to efficient over-the-air (OTA) aggregation in federated learning (FL) systems for distributed Internet-of-Things (IoT) scenarios. In this paper, we propose a novel manifold optimization formulation for low-rank model compression in FL that does not relax the low-rank constraint. Our optimization is conducted directly over the low-rank manifold, guaranteeing that the model is exactly low-rank. We also introduce a consensus penalty in the optimization formulation to support OTA aggregation. Based on our optimization formulation, we propose an alternating Riemannian optimization algorithm with a precoder that enables efficient OTA aggregation of low-rank local models without sacrificing training performance. Additionally, we provide convergence analysis in terms of key system parameters and conduct extensive experiments with real-world datasets to demonstrate the effectiveness of our proposed Riemannian low-rank model compression scheme compared to various state-of-the-art baselines.

What problem does this paper attempt to address?

The paper primarily addresses the issue of model compression in Federated Learning (FL), particularly the application of low-rank model compression in Over-the-Air (OTA) scenarios. Specifically, the paper tackles the following problems: 1. **Problems with existing model compression methods**: - Existing low-rank model compression methods typically use regularized nuclear norm penalties to approximate low-rank constraints. This approach is highly sensitive to the choice of hyperparameters, and the optimization process is non-convex and non-smooth, making it difficult to solve efficiently. - Current compression techniques are not directly applicable to efficient OTA aggregation in distributed Internet of Things (IoT) scenarios within Federated Learning. 2. **Proposed method**: - The paper proposes a novel manifold optimization formulation that directly optimizes on the low-rank manifold, ensuring that the model strictly meets low-rank requirements without the need for regularized nuclear norm penalties. - A consensus penalty term is introduced in the proposed optimization formulation to support OTA aggregation and ensure that local models do not deviate too far, thereby maintaining the low-rank structure of the global model after OTA aggregation. - Based on the above formulation, the paper also proposes an alternating Riemannian optimization algorithm that includes a pre-coder to achieve efficient OTA aggregation while ensuring the convergence of the algorithm. 3. **Main contributions**: - A novel manifold optimization formulation is proposed for low-rank model compression in Federated Learning environments, eliminating the need for convex relaxation and ensuring that the model maintains a low-rank structure throughout the training process. - An alternating Riemannian optimization algorithm with OTA aggregation is designed, introducing a Random Linear Coding (RLC) compression pre-coder to mitigate cross-term issues in OTA aggregation. - Theoretical analysis is provided, proving that under certain conditions, the proposed solution converges to a Karush-Kuhn-Tucker (KKT) point. - Extensive experiments validate that the proposed solution outperforms existing baseline methods in terms of compression performance and CPU time under a wide range of conditions. In summary, this paper aims to address the compatibility issue between model compression and OTA aggregation in Federated Learning. By proposing a novel optimization method and algorithm design, it effectively improves the efficiency of model compression and the overall performance of the Federated Learning system.

Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation

Efficient Wireless Federated Learning via Low-Rank Gradient Factorization

Secure Federated Learning with Model Compression.

Secure Federated Learning over Wireless Communication Networks with Model Compression

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

Efficient Model Compression for Hierarchical Federated Learning

Federated Learning With Lossy Distributed Source Coding: Analysis and Optimization

Federated Low-Rank Adaptation for Large Models Fine-Tuning over Wireless Networks

Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees

Federated Learning with Manifold Regularization and Normalized Update Reaggregation

Federated Learning in Multi-RIS-Aided Systems

FLoCoRA: Federated learning compression with low-rank adaptation

Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization

Over-the-Air Federated Learning and Optimization

FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications

Federated Learning via Over-the-Air Computation

Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data

FedFit: Server Aggregation Through Linear Regression in Federated Learning

Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression

Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation