Critical care: Estimating renal function in the ICU: even more challenging

S. Demirjian,E. Poggio

DOI: https://doi.org/10.1038/nrneph.2011.126

IF: 42.439

2011-09-06

Nature Reviews Nephrology

Abstract:

What problem does this paper attempt to address?

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Shaotian Yan,Xiang Tian,Rongxin Jiang,Yaowu Chen

DOI: https://doi.org/10.3390/app122211406

2022-01-01

Abstract:A small portion of mislabeled data can easily limit the performance of deep neural networks (DNNs) due to their high capacity for memorizing random labels. Thus, robust learning from noisy labels has become a key challenge for deep learning due to inadequate datasets with high-quality annotations. Most existing methods involve training models on clean sets by dividing clean samples from noisy ones, resulting in large amounts of mislabeled data being unused. To address this problem, we propose categorizing training samples into five fine-grained clusters based on the difficulty experienced by DNN models when learning them and label correctness. A novel fine-grained confidence modeling (FGCM) framework is proposed to cluster samples into these five categories; with each cluster, FGCM decides whether to accept the cluster data as they are, accept them with label correction, or accept them as unlabeled data. By applying different strategies to the fine-grained clusters, FGCM can better exploit training data than previous methods. Extensive experiments on widely used benchmarks CIFAR-10, CIFAR-100, clothing1M, and WebVision with different ratios and types of label noise demonstrate the superiority of our FGCM.
Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clients

Chenrui Wu,Zexi Li,Fangxin Wang,Chao Wu

2023-04-06

Abstract:Federated learning (FL) is a distributed framework for collaboratively training with privacy guarantees. In real-world scenarios, clients may have Non-IID data (local class imbalance) with poor annotation quality (label noise). The co-existence of label noise and class imbalance in FL's small local datasets renders conventional FL methods and noisy-label learning methods both ineffective. To address the challenges, we propose FedCNI without using an additional clean proxy dataset. It includes a noise-resilient local solver and a robust global aggregator. For the local solver, we design a more robust prototypical noise detector to distinguish noisy samples. Further to reduce the negative impact brought by the noisy samples, we devise a curriculum pseudo labeling method and a denoise Mixup training strategy. For the global aggregator, we propose a switching re-weighted aggregation method tailored to different learning periods. Extensive experiments demonstrate our method can substantially outperform state-of-the-art solutions in mix-heterogeneous FL environments.

Machine Learning,Artificial Intelligence
Collaboratively Learning Federated Models from Noisy Decentralized Data

Haoyuan Li,Mathias Funk,Nezihe Merve Gürel,Aaqib Saeed

2024-09-04

Abstract:Federated learning (FL) has emerged as a prominent method for collaboratively training machine learning models using local data from edge devices, all while keeping data decentralized. However, accounting for the quality of data contributed by local clients remains a critical challenge in FL, as local data are often susceptible to corruption by various forms of noise and perturbations, which compromise the aggregation process and lead to a subpar global model. In this work, we focus on addressing the problem of noisy data in the input space, an under-explored area compared to the label noise. We propose a comprehensive assessment of client input in the gradient space, inspired by the distinct disparity observed between the density of gradient norm distributions of models trained on noisy and clean input data. Based on this observation, we introduce a straightforward yet effective approach to identify clients with low-quality data at the initial stage of FL. Furthermore, we propose a noise-aware FL aggregation method, namely Federated Noise-Sifting (FedNS), which can be used as a plug-in approach in conjunction with widely used FL strategies. Our extensive evaluation on diverse benchmark datasets under different federated settings demonstrates the efficacy of FedNS. Our method effortlessly integrates with existing FL strategies, enhancing the global model's performance by up to 13.68% in IID and 15.85% in non-IID settings when learning from noisy decentralized data.

Machine Learning
Federated Learning with Instance-Dependent Noisy Label

Lei Wang,Jieming Bian,Jie Xu

2024-01-10

Abstract:Federated learning (FL) with noisy labels poses a significant challenge. Existing methods designed for handling noisy labels in centralized learning tend to lose their effectiveness in the FL setting, mainly due to the small dataset size and the heterogeneity of client data. While some attempts have been made to tackle FL with noisy labels, they primarily focused on scenarios involving class-conditional noise. In this paper, we study the more challenging and practical issue of instance-dependent noise (IDN) in FL. We introduce a novel algorithm called FedBeat (Federated Learning with Bayesian Ensemble-Assisted Transition Matrix Estimation). FedBeat aims to build a global statistically consistent classifier using the IDN transition matrix (IDNTM), which encompasses three synergistic steps: (1) A federated data extraction step that constructs a weak global model and extracts high-confidence data using a Bayesian model ensemble method. (2) A federated transition matrix estimation step in which clients collaboratively train an IDNTM estimation network based on the extracted data. (3) A federated classifier correction step that enhances the global model's performance by training it using a loss function tailored for noisy labels, leveraging the IDNTM. Experiments conducted on CIFAR-10 and SVHN verify that the proposed method significantly outperforms state-of-the-art methods.

Machine Learning,Computer Vision and Pattern Recognition
FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

Xinyuan Ji,Zhaowei Zhu,Wei Xi,Olga Gadyatskaya,Zilong Song,Yong Cai,Yang Liu

2024-03-25

Abstract:Federated Learning (FL) heavily depends on label quality for its performance. However, the label distribution among individual clients is always both noisy and heterogeneous. The high loss incurred by client-specific samples in heterogeneous label noise poses challenges for distinguishing between client-specific and noisy label samples, impacting the effectiveness of existing label noise learning approaches. To tackle this issue, we propose FedFixer, where the personalized model is introduced to cooperate with the global model to effectively select clean client-specific samples. In the dual models, updating the personalized model solely at a local level can lead to overfitting on noisy data due to limited samples, consequently affecting both the local and global models' performance. To mitigate overfitting, we address this concern from two perspectives. Firstly, we employ a confidence regularizer to alleviate the impact of unconfident predictions caused by label noise. Secondly, a distance regularizer is implemented to constrain the disparity between the personalized and global models. We validate the effectiveness of FedFixer through extensive experiments on benchmark datasets. The results demonstrate that FedFixer can perform well in filtering noisy label samples on different clients, especially in highly heterogeneous label noise scenarios.

Artificial Intelligence
Overhead-free Noise-tolerant Federated Learning: A New Baseline

Shiyi Lin,Deming Zhai,Feilong Zhang,Junjun Jiang,Xianming Liu,Xiangyang Ji

DOI: https://doi.org/10.1007/s11633-023-1449-1

2024-01-01

Machine Intelligence Research

Abstract:Federated learning (FL) is a promising decentralized machine learning approach that enables multiple distributed clients to train a model jointly while keeping their data private. However, in real-world scenarios, the supervised training data stored in local clients inevitably suffer from imperfect annotations, resulting in subjective, inconsistent and biased labels. These noisy labels can harm the collaborative aggregation process of FL by inducing inconsistent decision boundaries. Unfortunately, few attempts have been made towards noise-tolerant federated learning, with most of them relying on the strategy of transmitting overhead messages to assist noisy labels detection and correction, which increases the communication burden as well as privacy risks. In this paper, we propose a simple yet effective method for noise-tolerant FL based on the well-established co-training framework. Our method leverages the inherent discrepancy in the learning ability of the local and global models in FL, which can be regarded as two complementary views. By iteratively exchanging samples with their high confident predictions, the two models “teach each other” to suppress the influence of noisy labels. The proposed scheme enjoys the benefit of overhead cost-free and can serve as a robust and efficient baseline for noise-tolerant federated learning. Experimental results demonstrate that our method outperforms existing approaches, highlighting the superiority of our method.
FedDiv: Collaborative Noise Filtering for Federated Learning with Noisy Labels

Jichang Li,Guanbin Li,Hui Cheng,Zicheng Liao,Yizhou Yu

2024-02-16

Abstract:Federated learning with noisy labels (F-LNL) aims at seeking an optimal server model via collaborative distributed learning by aggregating multiple client models trained with local noisy or clean samples. On the basis of a federated learning framework, recent advances primarily adopt label noise filtering to separate clean samples from noisy ones on each client, thereby mitigating the negative impact of label noise. However, these prior methods do not learn noise filters by exploiting knowledge across all clients, leading to sub-optimal and inferior noise filtering performance and thus damaging training stability. In this paper, we present FedDiv to tackle the challenges of F-LNL. Specifically, we propose a global noise filter called Federated Noise Filter for effectively identifying samples with noisy labels on every client, thereby raising stability during local training sessions. Without sacrificing data privacy, this is achieved by modeling the global distribution of label noise across all clients. Then, in an effort to make the global model achieve higher performance, we introduce a Predictive Consistency based Sampler to identify more credible local data for local model training, thus preventing noise memorization and further boosting the training stability. Extensive experiments on CIFAR-10, CIFAR-100, and Clothing1M demonstrate that \texttt{FedDiv} achieves superior performance over state-of-the-art F-LNL methods under different label noise settings for both IID and non-IID data partitions. Source code is publicly available at <a class="link-external link-https" href="https://github.com/lijichang/FLNL-FedDiv" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

Zhen Qin,Shuiguang Deng,Mingyu Zhao,Xueqiang Yan

DOI: https://doi.org/10.1145/3580305.3599344

2023-01-01

Abstract:In cross-silo federated learning (FL), the data among clients are usually statistically heterogeneous (aka not independent and identically distributed, non-IID) due to diversified data sources, lowering the accuracy of FL. Although many personalized FL (PFL) approaches have been proposed to address this issue, they are only suitable for data with specific degrees of statistical heterogeneity. In the real world, the heterogeneity of data among clients is often immeasurable due to privacy concern, making the targeted selection of PFL approaches difficult. Besides, in cross-silo FL, clients are usually from different organizations, tending to hold architecturally different private models. In this work, we propose a novel FL framework, FedAPEN, which combines mutual learning and ensemble learning to take the advantages of private and shared global models while allowing heterogeneous models. Within FedAPEN, we propose two mechanisms to coordinate and promote model ensemble such that FedAPEN achieves excellent accuracy on various data distributions without prior knowledge of data heterogeneity, and thus, obtains the adaptability to data heterogeneity. We conduct extensive experiments on four real-world datasets, including: 1) Fashion MNIST, CIFAR-10, and CIFAR-100, each with ten different types and degrees of label distribution skew; and 2) eICU with feature distribution skew. The experiments demonstrate that FedAPEN almost obtains superior accuracy on data with varying types and degrees of heterogeneity compared with baselines.
Medical federated learning with joint graph purification for noisy label learning

Zhen Chen,Wuyang Li,Xiaohan Xing,Yixuan Yuan

DOI: https://doi.org/10.1016/j.media.2023.102976

IF: 10.9

2023-12-01

Medical Image Analysis

Abstract:In terms of increasing privacy issues, Federated Learning (FL) has received extensive attention in medical imaging. Through collaborative training, FL can produce superior diagnostic models with global knowledge, while preserving private data locally. In practice, medical diagnosis suffers from intra-/inter-observer variability, thus label noise is inevitable in dataset preparation. Different from existing studies on centralized datasets, the label noise problem in FL scenarios confronts more challenges, due to data inaccessibility and even noise heterogeneity. In this work, we propose a federated framework with joint Graph Purification (FedGP) to address the label noise in FL through server and clients collaboration. Specifically, to overcome the impact of label noise on local training, we first devise a noisy graph purification on the client side to generate reliable pseudo labels by progressively expanding the purified graph with topological knowledge. Then, we further propose a graph-guided negative ensemble loss to exploit the topology of the client-side purified graph with robust complementary supervision against label noise. Moreover, to address the FL label noise with data silos, we propose a global centroid aggregation on the server side to produce a robust classifier with global knowledge, which can be optimized collaboratively in the FL framework. Extensive experiments are conducted on endoscopic and pathological images with the comparison under the homogeneous, heterogeneous, and real-world label noise for medical FL. Among these diverse noisy FL settings, our FedGP framework significantly outperforms denoising and noisy FL state-of-the-arts by a large margin. The source code is available at https://github.com/CUHK-AIM-Group/FedGP.

engineering, biomedical,computer science, interdisciplinary applications, artificial intelligence,radiology, nuclear medicine & medical imaging
FedRGL: Robust Federated Graph Learning for Label Noise

De Li,Haodong Qian,Qiyu Li,Zhou Tan,Zemin Gan,Jinyan Wang,Xianxian Li

2024-11-28

Abstract:Federated Graph Learning (FGL) is a distributed machine learning paradigm based on graph neural networks, enabling secure and collaborative modeling of local graph data among clients. However, label noise can degrade the global model's generalization performance. Existing federated label noise learning methods, primarily focused on computer vision, often yield suboptimal results when applied to FGL. To address this, we propose a robust federated graph learning method with label noise, termed FedRGL. FedRGL introduces dual-perspective consistency noise node filtering, leveraging both the global model and subgraph structure under class-aware dynamic thresholds. To enhance client-side training, we incorporate graph contrastive learning, which improves encoder robustness and assigns high-confidence pseudo-labels to noisy nodes. Additionally, we measure model quality via predictive entropy of unlabeled nodes, enabling adaptive robust aggregation of the global model. Comparative experiments on multiple real-world graph datasets show that FedRGL outperforms 12 baseline methods across various noise rates, types, and numbers of clients.

Machine Learning
Data Quality-Aware Client Selection in Heterogeneous Federated Learning

Shinan Song,Yaxin Li,Jin Wan,Xianghua Fu,Jingyan Jiang

DOI: https://doi.org/10.3390/math12203229

IF: 2.4

2024-10-16

Mathematics

Abstract:Federated Learning (FL) enables decentralized data utilization while maintaining edge user privacy, but it faces challenges due to statistical heterogeneity. Existing approaches address client drift and data heterogeneity issues. However, real-world settings often involve low-quality data with noisy features, such as covariate drift or adversarial samples, which are usually ignored. Noisy samples significantly impact the global model's accuracy and convergence rate. Assessing data quality and selectively aggregating updates from high-quality clients is crucial, but dynamically perceiving data quality without additional computations or data exchanges is challenging. In this paper, we introduce the FedDQA (Federated learning via Data Quality-Aware) (FedDQA) framework. We discover increased data noise leads to slower loss reduction during local model training. We propose a loss sharpness-based Data-Quality-Awareness (DQA) metric to differentiate between high-quality and low-quality data. Based on the DQA, we design a client selection algorithm that strategically selects participant clients to reduce the negative impact of noisy clients. Experiment results indicate that FedDQA significantly outperforms the baselines. Notably, it achieves up to a 4% increase in global model accuracy and demonstrates faster convergence rates.

mathematics
Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels

Yuxin Tian,Mouxing Yang,Yuhao Zhou,Jian Wang,Qing Ye,Tongliang Liu,Gang Niu,Jiancheng Lv

2024-11-30

Abstract:The success of most federated learning (FL) methods heavily depends on label quality, which is often inaccessible in real-world scenarios, such as medicine, leading to the federated label-noise (F-LN) problem. In this study, we observe that the global model of FL memorizes the noisy labels slowly. Based on the observations, we propose a novel approach dubbed Global Reviser for Federated Learning with Noisy Labels (FedGR) to enhance the label-noise robustness of FL. In brief, FedGR employs three novel modules to achieve noisy label sniffing and refining, local knowledge revising, and local model regularization. Specifically, the global model is adopted to infer local data proxies for global sample selection and refine incorrect labels. To maximize the utilization of local knowledge, we leverage the global model to revise the local exponential moving average (EMA) model of each client and distill it into the clients' models. Additionally, we introduce a global-to-local representation regularization to mitigate the overfitting of noisy labels. Extensive experiments on three F-LNL benchmarks against seven baseline methods demonstrate the effectiveness of the proposed FedGR.

Machine Learning,Computer Vision and Pattern Recognition
FedMT: Federated Learning with Mixed-type Labels

Qiong Zhang,Jing Peng,Xin Zhang,Aline Talhouk,Gang Niu,Xiaoxiao Li

2024-02-16

Abstract:In federated learning (FL), classifiers (e.g., deep networks) are trained on datasets from multiple data centers without exchanging data across them, which improves the sample efficiency. However, the conventional FL setting assumes the same labeling criterion in all data centers involved, thus limiting its practical utility. This limitation becomes particularly notable in domains like disease diagnosis, where different clinical centers may adhere to different standards, making traditional FL methods unsuitable. This paper addresses this important yet under-explored setting of FL, namely FL with mixed-type labels, where the allowance of different labeling criteria introduces inter-center label space differences. To address this challenge effectively and efficiently, we introduce a model-agnostic approach called FedMT, which estimates label space correspondences and projects classification scores to construct loss functions. The proposed FedMT is versatile and integrates seamlessly with various FL methods, such as FedAvg. Experimental results on benchmark and medical datasets highlight the substantial improvement in classification accuracy achieved by FedMT in the presence of mixed-type labels.

Machine Learning,Artificial Intelligence,Distributed, Parallel, and Cluster Computing
Tackling Noisy Clients in Federated Learning with End-to-end Label Correction

Xuefeng Jiang,Sheng Sun,Jia Li,Jingjing Xue,Runhan Li,Zhiyuan Wu,Gang Xu,Yuwei Wang,Min Liu

DOI: https://doi.org/10.1145/3627673.3679550

2024-08-08

Abstract:Recently, federated learning (FL) has achieved wide successes for diverse privacy-sensitive applications without sacrificing the sensitive private information of clients. However, the data quality of client datasets can not be guaranteed since corresponding annotations of different clients often contain complex label noise of varying degrees, which inevitably causes the performance degradation. Intuitively, the performance degradation is dominated by clients with higher noise rates since their trained models contain more misinformation from data, thus it is necessary to devise an effective optimization scheme to mitigate the negative impacts of these noisy clients. In this work, we propose a two-stage framework FedELC to tackle this complicated label noise issue. The first stage aims to guide the detection of noisy clients with higher label noise, while the second stage aims to correct the labels of noisy clients' data via an end-to-end label correction framework which is achieved by learning possible ground-truth labels of noisy clients' datasets via back propagation. We implement sixteen related methods and evaluate five datasets with three types of complicated label noise scenarios for a comprehensive comparison. Extensive experimental results demonstrate our proposed framework achieves superior performance than its counterparts for different scenarios. Additionally, we effectively improve the data quality of detected noisy clients' local datasets with our label correction framework. The code is available at <a class="link-external link-https" href="https://github.com/Sprinter1999/FedELC" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence
Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Xiaotian Yu,Shengxuming Zhang,Lingxiang Jia,Yuexuan Wang,Mingli Song,Zunlei Feng

DOI: https://doi.org/10.1016/j.neucom.2023.126829

IF: 6

2024-01-01

Neurocomputing

Abstract:Recently, many benchmark datasets have been found to contain noisy labels caused by unavoidable human mistakes. Many researchers propose new noise-aware loss functions to achieve robust classification performance. However, we discover that existing noise-aware loss functions cannot fully heal the damage caused by the noise. On the other hand, some methods filter out low confidence samples and train new models, whereas the filtered samples contain both noisy and hard samples that are critical for the robustness of models. Based on the above two discoveries, we devised the Noise-aware Network (NA-Net) for robust training with noisy labels. Each layer of NA-Net contains three groups of convolution kernels responsible for mix samples, clean samples, and noisy samples, termed as mix-kernels, clean-kernels, and noise-kernels, respectively. Mix-kernels are used for finding the clean samples with a newly devised noise-immune (NI) loss function; clean-kernels are targeted at learning better features without being misguided by noise; noise-kernels are trained by the remaining samples to rectify wrong labels for the next iteration. Meanwhile, for increasing the classification performance of mix-kernels, the extracted feature maps of clean-kernels without being poisoned are combined as the input of mix-kernels of the next layer. Also, the knowledge distillation strategy is adopted to distill the knowledge from clean-kernels to the noise-kernels. Extensive experiments demonstrate that the mutual promotion of three groups of kernels in NA-Net achieves state-of-the-art performance on both artificial noisy datasets and real-world datasets.
An Edge-Assisted Federated Contrastive Learning Method with Local Intrinsic Dimensionality in Noisy Label Environment

Siyuan Wu,Guoming Zhang,Fei Dai,Bowen Liu,Wanchun Dou

DOI: https://doi.org/10.1002/spe.3295

2024-01-01

Abstract:The advent of federated learning (FL) has presented a viable solution for distributed training in edge environment, while simultaneously ensuring the preservation of privacy. In real-world scenarios, edge devices may be subject to label noise caused by environmental differences, automated weakly supervised annotation, malicious tampering, or even human error. However, the potential of the noisy samples have not been fully leveraged by prior studies on FL aimed at addressing label noise. Rather, they have primarily focused on conventional filtering or correction techniques to alleviate the impact of noisy labels. To tackle this challenge, a method, named DETECTION, is proposed in this article. It aims at effectively detecting noisy clients and mitigating the adverse impact of label noise while preserving data privacy. Specially, a confidence scoring mechanism based on local intrinsic dimensionality (LID) is investigated for distinguishing noisy clients from clean clients. Then, a loss function based on prototype contrastive learning is designed to optimize the local model. To address the varying levels of noise across clients, a LID weighted aggregation strategy (LA) is introduced. Experimental results on three datasets demonstrate the effectiveness of DETECTION in addressing the issue of label noise in FL while maintaining data privacy.
FedNoisy: Federated Noisy Label Learning Benchmark

Siqi Liang,Jintao Huang,Junyuan Hong,Dun Zeng,Jiayu Zhou,Zenglin Xu

2024-05-30

Abstract:Federated learning has gained popularity for distributed learning without aggregating sensitive data from clients. But meanwhile, the distributed and isolated nature of data isolation may be complicated by data quality, making it more vulnerable to noisy labels. Many efforts exist to defend against the negative impacts of noisy labels in centralized or federated settings. However, there is a lack of a benchmark that comprehensively considers the impact of noisy labels in a wide variety of typical FL settings. In this work, we serve the first standardized benchmark that can help researchers fully explore potential federated noisy settings. Also, we conduct comprehensive experiments to explore the characteristics of these data settings and unravel challenging scenarios on the federated noisy label learning, which may guide method development in the future. We highlight the 20 basic settings for more than 5 datasets proposed in our benchmark and standardized simulation pipeline for federated noisy label learning. We hope this benchmark can facilitate idea verification in federated learning with noisy labels. \texttt{FedNoisy} is available at \codeword{<a class="link-external link-https" href="https://github.com/SMILELab-FL/FedNoisy" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Artificial Intelligence
FedVal: Different good or different bad in federated learning

Viktor Valadi,Xinchi Qiu,Pedro Porto Buarque de Gusmão,Nicholas D. Lane,Mina Alibeigi

2023-06-07

Abstract:Federated learning (FL) systems are susceptible to attacks from malicious actors who might attempt to corrupt the training model through various poisoning attacks. FL also poses new challenges in addressing group bias, such as ensuring fair performance for different demographic groups. Traditional methods used to address such biases require centralized access to the data, which FL systems do not have. In this paper, we present a novel approach FedVal for both robustness and fairness that does not require any additional information from clients that could raise privacy concerns and consequently compromise the integrity of the FL system. To this end, we propose an innovative score function based on a server-side validation method that assesses client updates and determines the optimal aggregation balance between locally-trained models. Our research shows that this approach not only provides solid protection against poisoning attacks but can also be used to reduce group bias and subsequently promote fairness while maintaining the system's capability for differential privacy. Extensive experiments on the CIFAR-10, FEMNIST, and PUMS ACSIncome datasets in different configurations demonstrate the effectiveness of our method, resulting in state-of-the-art performances. We have proven robustness in situations where 80% of participating clients are malicious. Additionally, we have shown a significant increase in accuracy for underrepresented labels from 32% to 53%, and increase in recall rate for underrepresented features from 19% to 50%.

Machine Learning,Artificial Intelligence,Cryptography and Security
Federated Learning with Extremely Noisy Clients via Negative Distillation

Yang Lu,Lin Chen,Yonggang Zhang,Yiliang Zhang,Bo Han,Yiu-ming Cheung,Hanzi Wang

2024-02-23

Abstract:Federated learning (FL) has shown remarkable success in cooperatively training deep models, while typically struggling with noisy labels. Advanced works propose to tackle label noise by a re-weighting strategy with a strong assumption, i.e., mild label noise. However, it may be violated in many real-world FL scenarios because of highly contaminated clients, resulting in extreme noise ratios, e.g., $>$90%. To tackle extremely noisy clients, we study the robustness of the re-weighting strategy, showing a pessimistic conclusion: minimizing the weight of clients trained over noisy data outperforms re-weighting strategies. To leverage models trained on noisy clients, we propose a novel approach, called negative distillation (FedNed). FedNed first identifies noisy clients and employs rather than discards the noisy clients in a knowledge distillation manner. In particular, clients identified as noisy ones are required to train models using noisy labels and pseudo-labels obtained by global models. The model trained on noisy labels serves as a `bad teacher' in knowledge distillation, aiming to decrease the risk of providing incorrect information. Meanwhile, the model trained on pseudo-labels is involved in model aggregation if not identified as a noisy client. Consequently, through pseudo-labeling, FedNed gradually increases the trustworthiness of models trained on noisy clients, while leveraging all clients for model aggregation through negative distillation. To verify the efficacy of FedNed, we conduct extensive experiments under various settings, demonstrating that FedNed can consistently outperform baselines and achieve state-of-the-art performance. Our code is available at <a class="link-external link-https" href="https://github.com/linChen99/FedNed" rel="external noopener nofollow">this https URL</a>.

Machine Learning
Robust Learning under Hybrid Noise

Yang Wei,Shuo Chen,Shanshan Ye,Bo Han,Chen Gong

2024-07-05

Abstract:Feature noise and label noise are ubiquitous in practical scenarios, which pose great challenges for training a robust machine learning model. Most previous approaches usually deal with only a single problem of either feature noise or label noise. However, in real-world applications, hybrid noise, which contains both feature noise and label noise, is very common due to the unreliable data collection and annotation processes. Although some results have been achieved by a few representation learning based attempts, this issue is still far from being addressed with promising performance and guaranteed theoretical analyses. To address the challenge, we propose a novel unified learning framework called "Feature and Label Recovery" (FLR) to combat the hybrid noise from the perspective of data recovery, where we concurrently reconstruct both the feature matrix and the label matrix of input data. Specifically, the clean feature matrix is discovered by the low-rank approximation, and the ground-truth label matrix is embedded based on the recovered features with a nuclear norm regularization. Meanwhile, the feature noise and label noise are characterized by their respective adaptive matrix norms to satisfy the corresponding maximum likelihood. As this framework leads to a non-convex optimization problem, we develop the non-convex Alternating Direction Method of Multipliers (ADMM) with the convergence guarantee to solve our learning objective. We also provide the theoretical analysis to show that the generalization error of FLR can be upper-bounded in the presence of hybrid noise. Experimental results on several typical benchmark datasets clearly demonstrate the superiority of our proposed method over the state-of-the-art robust learning approaches for various noises.

Machine Learning

Critical care: Estimating renal function in the ICU: even more challenging

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clients

Collaboratively Learning Federated Models from Noisy Decentralized Data

Federated Learning with Instance-Dependent Noisy Label

FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

Overhead-free Noise-tolerant Federated Learning: A New Baseline

FedDiv: Collaborative Noise Filtering for Federated Learning with Noisy Labels

FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

Medical federated learning with joint graph purification for noisy label learning

FedRGL: Robust Federated Graph Learning for Label Noise

Data Quality-Aware Client Selection in Heterogeneous Federated Learning

Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels

FedMT: Federated Learning with Mixed-type Labels

Tackling Noisy Clients in Federated Learning with End-to-end Label Correction

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

An Edge-Assisted Federated Contrastive Learning Method with Local Intrinsic Dimensionality in Noisy Label Environment

FedNoisy: Federated Noisy Label Learning Benchmark

FedVal: Different good or different bad in federated learning

Federated Learning with Extremely Noisy Clients via Negative Distillation

Robust Learning under Hybrid Noise