Abstract:Federated learning enables multiple decentralized clients to learn collaboratively without sharing the local training data. However, the expensive annotation cost to acquire data labels on local clients remains an obstacle in utilizing local data. In this paper, we propose a federated active learning paradigm to efficiently learn a global model with limited annotation budget while protecting data privacy in a decentralized learning way. The main challenge faced by federated active learning is the mismatch between the active sampling goal of the global model on the server and that of the asynchronous local clients. This becomes even more significant when data is distributed non-IID across local clients. To address the aforementioned challenge, we propose Knowledge-Aware Federated Active Learning (KAFAL), which consists of Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory Federated Update (KCFU). KSAS is a novel active sampling method tailored for the federated active learning problem. It deals with the mismatch challenge by sampling actively based on the discrepancies between local and global models. KSAS intensifies specialized knowledge in local clients, ensuring the sampled data to be informative for both the local clients and the global model. KCFU, in the meantime, deals with the client heterogeneity caused by limited data and non-IID data distributions. It compensates for each client's ability in weak classes by the assistance of the global model. Extensive experiments and analyses are conducted to show the superiority of KSAS over the state-of-the-art active learning methods and the efficiency of KCFU under the federated active learning framework.

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve This paper aims to address the issue of how to efficiently conduct Federated Active Learning (FAL) under a Non-Independent and Identically Distributed (Non-IID) data environment with a limited annotation budget while protecting data privacy. ### Detailed Interpretation #### Background and Challenges 1. **Federated Learning (FL)**: - Federated Learning is a decentralized learning paradigm that allows multiple dispersed clients to collaboratively learn without sharing local data, ultimately aggregating a powerful global model on a central server. - However, high annotation costs remain a major obstacle to utilizing local data, especially in scenarios where large-scale medical data is distributed across different medical institutions, each with very limited data annotation experts. 2. **Federated Active Learning (FAL)**: - The goal of Federated Active Learning is to efficiently train a global model by actively selecting the most informative data for annotation within a limited annotation budget. - The main challenge lies in the mismatch of active sampling targets between clients and the server, which is particularly pronounced in Non-IID data distributions. #### Proposed Method 1. **Knowledge-Aware Federated Active Learning (KAFAL)**: - KAFAL consists of two key components: Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory Federated Update (KCFU). 2. **Knowledge-Specialized Active Sampling (KSAS)**: - KSAS is a novel active sampling strategy designed to select universally informative data for both clients and the global model by computing the reinforced differences between the outputs of the clients and the global model. - Specifically, KSAS reinforces the proprietary knowledge of clients, ensuring that the sampled data is informative for both local clients and the global model. 3. **Knowledge-Compensatory Federated Update (KCFU)**: - KCFU addresses client heterogeneity caused by limited data and Non-IID data by compensating for each client's deficiencies in weak categories through knowledge distillation from the global model. - KCFU achieves this by introducing an additional loss term \( L_{\text{compen}} \), which utilizes unlabeled data for knowledge distillation. #### Experimental Results 1. **Datasets**: - The authors conducted experiments on three image classification datasets: CIFAR10, CIFAR100, and MNIST. - They also applied the method in a more realistic scenario, using the NIH Chest X-ray dataset for medical image classification. 2. **Experimental Setup**: - ResNet-8 was used as the model architecture for both the server and clients. - In each communication round, 80% of the clients were randomly selected for local updates. - Data distribution was generated using a Dirichlet distribution to create Non-IID data. 3. **Experimental Results**: - KAFAL achieved state-of-the-art results on both CIFAR10 and CIFAR100. - As the amount of annotated data increased, the gap between KAFAL and other baseline methods gradually widened. - On CIFAR10, KAFAL outperformed methods such as BADGE, Entropy, Margin, and Core-set by approximately 3%. - On CIFAR100, due to the 10-fold increase in the number of categories, the task became more challenging, and KAFAL's advantage was relatively smaller. ### Conclusion The paper proposes a

Knowledge-Aware Federated Active Learning with Non-IID Data

A Federated Learning Framework Via Decentralized Data Valuation for Chronic Disease Healthcare

MCKD: Mutually Collaborative Knowledge Distillation for Federated Domain Adaptation and Generalization

Distributed Active Learning.

Active Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning

Re-thinking Federated Active Learning based on Inter-class Diversity

Federated Active Learning (F-AL): An Efficient Annotation Strategy for Federated Learning

Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

Fedadkd:heterogeneous federated learning via adaptive knowledge distillation

Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor

Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts

Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement

FLAS: Computation and Communication Efficient Federated Learning via Adaptive Sampling

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

Federated Active Learning for Target Domain Generalisation

Active-Passive Federated Learning for Vertically Partitioned Multi-view Data

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

Towards Personalized Federated Learning via Comprehensive Knowledge Distillation