Abstract:In distributed learning systems, robustness threat may arise from two major sources. On the one hand, due to distributional shifts between training data and test data, the trained model could exhibit poor out-of-sample performance. On the other hand, a portion of working nodes might be subject to Byzantine attacks, which could invalidate the learning result. In this article, we propose a new research direction that jointly considers distributional shifts and Byzantine attacks. We illuminate the major challenges in addressing these two issues simultaneously. Accordingly, we design a new algorithm that equips distributed learning with both distributional robustness and Byzantine robustness. Our algorithm is built on recent advances in distributionally robust optimization (DRO) as well as norm-based screening (NBS), a robust aggregation scheme against Byzantine attacks. We provide convergence proofs in three cases of the learning model being nonconvex, convex, and strongly convex for the proposed algorithm, shedding light on its convergence behaviors and endurability against Byzantine attacks. In particular, we deduce that any algorithm employing NBS (including ours) cannot converge when the percentage of Byzantine nodes is [Formula: see text] or higher, instead of [Formula: see text] , which is the common belief in current literature. The experimental results verify our theoretical findings (on the breakpoint of NBS and others) and also demonstrate the effectiveness of our algorithm against both robustness issues, justifying our choice of NBS over other widely used robust aggregation schemes. To the best of our knowledge, this is the first work to address distributional shifts and Byzantine attacks simultaneously.

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

Byzantine-robust Variance-Reduced Federated Learning over Distributed Non-I.i.d. Data

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Byzantine-robust decentralized stochastic optimization with stochastic gradient noise-independent learning error

Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environment

Adaptive Distributed Learning with Byzantine Robustness: A Gradient-Projection-Based Method

Byzantine-Robust Aggregation with Gradient Difference Compression and Stochastic Variance Reduction for Federated Learning

High-dimensional M-estimation for Byzantine-robust Decentralized Learning

Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences

Byzantine-Robust Distributed Learning with Compression.

Distributed Online Learning with Adversarial Participants in an Adversarial Environment

Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning

Byzantine-Robust Stochastic Gradient Descent for Distributed Low-Rank Matrix Completion

Privacy-Preserving Aggregation for Decentralized Learning with Byzantine-Robustness

Self-Driven Entropy Aggregation for Byzantine-Robust Heterogeneous Federated Learning

Resilient to byzantine attacks finite-sum optimization over networks

Buffered Asynchronous SGD for Byzantine Learning

High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers