Abstract:Traditional federated learning uses the number of samples to calculate the weights of each client model and uses this fixed weight value to fusion the global model. However, in practical scenarios, each client's device and data heterogeneity leads to differences in the quality of each client's model. Thus the contribution to the global model is not wholly determined by the sample size. In addition, if clients intentionally upload low-quality or malicious models, using these models for aggregation will lead to a severe decrease in global model accuracy. Traditional federated learning algorithms do not address these issues. To solve this probelm, we propose FedDRL, a model fusion approach using reinforcement learning based on a two staged approach. In the first stage, Our method could filter out malicious models and selects trusted client models to participate in the model fusion. In the second stage, the FedDRL algorithm adaptively adjusts the weights of the trusted client models and aggregates the optimal global model. We also define five model fusion scenarios and compare our method with two baseline algorithms in those scenarios. The experimental results show that our algorithm has higher reliability than other algorithms while maintaining accuracy.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve two main challenges in Federated Learning (FL): 1. **The problem of integrating malicious models**: - In traditional federated learning, models of all participants are usually integrated by weighted averaging. However, if some participants submit malicious models, these low - quality or harmful models will seriously affect the accuracy and performance of the global model. - The paper proposes a method to identify and exclude these malicious models to ensure the reliability of the global model. 2. **The problem of device and data heterogeneity**: - Due to the heterogeneity of different devices and datasets (i.e., Non - IID data distribution), simply assigning weights according to the number of samples cannot reflect the true quality of the model. - An adaptive method is required to dynamically adjust the integration weight of each model to improve the performance of the global model. To solve these problems, the authors propose a trust - based federated learning model fusion method (FedDRL) using phased reinforcement learning. Specifically, FedDRL includes two phases: - **Phase 1: A reliable client selection mechanism**: - Use reinforcement learning techniques to dynamically screen out trustworthy clients and exclude those participants who may submit malicious models. - **Phase 2: An adaptive model fusion method**: - Dynamically assign weights according to the quality of the model to ensure that high - quality models have a greater impact on the global model, thereby improving the overall performance of the global model. Through these two phases, FedDRL not only improves the accuracy of the global model but also enhances the reliability and robustness of the system. ### Formula representation - **The model fusion formula in traditional federated learning**: \[ \theta_{\text{global}}=\sum_{i = 1}^{N}w_i\theta_i,\quad\text{where}\quad w_i=\frac{S_i}{\sum_{i = 1}^{N}S_i},\quad w_i\geq0,\quad\sum_{i = 1}^{N}w_i = 1 \] where \( \theta_i \) is the model parameter of the \( i \) - th client and \( S_i \) is the number of data samples of the \( i \) - th client. - **The adaptive weight calculation formula**: \[ W_t=\left[w_t^1,w_t^2,\ldots,w_t^k\right] \] where \( w_t^i \) is the weight value output by the agent for the \( i \) - th model at the \( t \) - th iteration. - **The global model fusion formula**: \[ \theta_{\text{global}}^k=\sum_{t = 1}^{T}w_t^i\theta_i \] ### Summary FedDRL effectively solves the challenges brought by malicious model fusion and data heterogeneity in federated learning by introducing a phased reinforcement learning method, and improves the accuracy and reliability of the global model.

FedDRL: A Trustworthy Federated Learning Model Fusion Method Based on Staged Reinforcement Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

An Optimization Method for Non-IID Federated Learning Based on Deep Reinforcement Learning

Dynamic Fair Federated Learning Based on Reinforcement Learning

FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients

Ensemble Distillation for Robust Model Fusion in Federated Learning

Model Fusion from Unauthorized Clients in Federated Learning

FedDdrl: Federated Double Deep Reinforcement Learning for Heterogeneous IoT with Adaptive Early Client Termination and Local Epoch Adjustment

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

FedMD: Heterogenous Federated Learning via Model Distillation

Optimizing Hierarchical Federated Learning: A Reinforcement Learning Approach

R $^{2}$ Fed: Resilient Reinforcement Federated Learning for Industrial Applications

FedAA: A Reinforcement Learning Perspective on Adaptive Aggregation for Fair and Robust Federated Learning

Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination

FedRAD: Heterogeneous Federated Learning via Relational Adaptive Distillation

Fedadkd:heterogeneous federated learning via adaptive knowledge distillation

Fusion of Global and Local Knowledge for Personalized Federated Learning

FRL: Federated Rank Learning

A Fair Federated Learning Framework with Reinforcement Learning.

Ranking-based Client Selection with Imitation Learning for Efficient Federated Learning