Abstract:In this paper, we address the challenge of certifying the performance of a machine learning model on an unseen target network, using measurements from an available source network. We focus on a scenario where heterogeneous datasets are distributed across a source network of clients, all connected to a central server. Specifically, consider a source network "A" composed of $K$ clients, each holding private data from unique and heterogeneous distributions, which are assumed to be independent samples from a broader meta-distribution $\mu$. Our goal is to provide certified guarantees for the model's performance on a different, unseen target network "B," governed by another meta-distribution $\mu'$, assuming the deviation between $\mu$ and $\mu'$ is bounded by either the Wasserstein distance or an $f$-divergence. We derive theoretical guarantees for the model's empirical average loss and provide uniform bounds on the risk CDF, where the latter correspond to novel and adversarially robust versions of the Glivenko-Cantelli theorem and the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality. Our bounds are computable in polynomial time with a polynomial number of queries to the $K$ clients, preserving client privacy by querying only the model's (potentially adversarial) loss on private data. We also establish non-asymptotic generalization bounds that consistently converge to zero as both $K$ and the minimum client sample size grow. Extensive empirical evaluations validate the robustness and practicality of our bounds across real-world tasks.

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is: **How to use measurement data from the source network to provide certification guarantees for the performance of machine - learning models on an unseen target network in a federated learning environment**. Specifically, the paper focuses on how to evaluate and guarantee the performance of machine - learning models on an unseen target network when heterogeneous data sets are distributed among client networks connected to a central server. ### Problem Background In Federated Learning (FL), training data and computing resources are scattered across various clients in the network, which brings several challenges. Especially when there is heterogeneity in the data distribution of different clients, evaluating the performance of a trained model on unseen clients or networks becomes particularly complex. For example, a model trained on data in one city may need to be applied to clients in another city, and there may be differences in the data distribution between these two cities. ### Specific Problems The paper aims to solve the following specific problems: 1. **Model Evaluation under Heterogeneous Data Distribution**: How to evaluate the performance of a model on unseen clients or networks in a federated learning environment. 2. **Generalization Ability across Networks**: How to provide theoretical guarantees for the performance of a model under different meta - distributions, especially when there is a deviation between the meta - distribution of the target network and that of the source network. 3. **Privacy Protection**: How to evaluate the performance of a model by querying its loss without directly accessing the private data of clients. ### Solutions The paper proposes a method to estimate the performance of a model on the target network by sampling the losses of clients in the source network and combining metrics such as Wasserstein distance or f - divergence. Specifically: - **Theoretical Guarantees**: The paper derives theoretical guarantees for the empirical average loss of the model and provides uniform bounds for the cumulative distribution function (CDF) of risk. These bounds correspond to new adversarial - robust versions of the Glivenko - Cantelli theorem and the Dvoretzky - Kiefer - Wolfowitz (DKW) inequality. - **Computational Efficiency**: These bounds can be calculated in polynomial time and only require a polynomial number of queries to K clients, thus protecting client privacy. - **Non - Asymptotic Generalization Bounds**: The paper also establishes non - asymptotic generalization bounds, which converge uniformly to zero as the number of clients K and the minimum client sample size increase. ### Application Scenarios This method is applicable to various real - world tasks, such as personalized recommendation systems in mobile networks, analysis of medical and health data, etc. It can effectively evaluate the performance of a model on unseen clients or networks and ensure the robustness and practicality of the model. In summary, this paper solves the problem of performance evaluation and guarantee of models on unseen networks in federated learning by introducing new theoretical frameworks and technical means, which has important theoretical and practical significance.

Robust Model Evaluation over Large-scale Federated Networks

Robust Federated Learning in a Heterogeneous Environment.

Federated Learning With Unreliable Clients: Performance Analysis and Mechanism Design

Globally-Robust Neural Networks

On the Necessity of Collaboration for Online Model Selection with Decentralized Data

Towards Model-Agnostic Federated Learning over Networks

From Learning to Analytics: Improving Model Efficacy with Goal-Directed Client Selection

Measuring Robustness of Deep Neural Networks from the Lens of Statistical Model Checking.

Learning to Generalize in Heterogeneous Federated Networks

Robust Federated Learning for Heterogeneous Model and Data

The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness

Robust Aggregation for Federated Learning

Robust Validation: Confident Predictions Even When Distributions Shift

Certifiably Byzantine-Robust Federated Conformal Prediction

Robust Model Aggregation for Heterogeneous Federated Learning: Analysis and Optimizations

Certified Adversarial Robustness Under the Bounded Support Set.

On the Certified Robustness for Ensemble Models and Beyond

Federated Learning Aggregation: New Robust Algorithms with Guarantees

SureFED: Robust Federated Learning via Uncertainty-Aware Inward and Outward Inspection

Federated Calibration and Evaluation of Binary Classifiers

Towards More Scalable and Robust Machine Learning