Federated Prompt Learning for Weather Foundation Models on Devices

Shengchao Chen,Guodong Long,Tao Shen,Jing Jiang,Chengqi Zhang

2024-04-22

Abstract:On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing, holds significance for supporting human activates. Federated Learning is a promising solution for such forecasting by enabling collaborative model training without sharing raw data. However, it faces three main challenges that hinder its reliability: (1) data heterogeneity among devices due to geographic differences; (2) data homogeneity within individual devices and (3) communication overload from sending large model parameters for collaboration. To address these challenges, this paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD), which enables devices to obtain highly customized models while maintaining communication efficiency. Concretely, our Adaptive Prompt Tuning leverages lightweight prompts guide frozen foundation model to generate more precise predictions, also conducts prompt-based multi-level communication to encourage multi-source knowledge fusion and regulate optimization. Additionally, Dynamic Graph Modeling constructs graphs from prompts, prioritizing collaborative training among devices with similar data distributions to against heterogeneity. Extensive experiments demonstrates FedPoD leads the performance among state-of-the-art baselines across various setting in real-world on-device weather forecasting datasets.

Machine Learning

What problem does this paper attempt to address?

The paper aims to address several key issues in on-device weather forecasting:

1. **Data Heterogeneity**: Significant differences in data across different devices due to geographical location variations.
2. **Data Homogeneity**: High similarity of data within a single device, which affects the effectiveness of personalized models.
3. **Communication Burden**: Sending a large number of model parameters for collaborative training in federated learning increases the communication burden.

To tackle these challenges, the authors propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD), which achieves efficient communication while maintaining highly customized models through the following two main components:
- **Adaptive Prompt Tuning**: Utilizes lightweight prompts to guide the frozen foundation model to generate more accurate predictions and conducts multi-level communication to facilitate multi-source knowledge fusion.
- **Dynamic Graph Modeling**: Constructs a dynamic graph based on geographical information, prioritizing collaborative training among devices with similar data distributions, thereby mitigating the impact of data heterogeneity.

Experimental results show that FedPoD performs excellently in various benchmarks, particularly in reducing communication load and protecting privacy.