Abstract:Distributed learning has become a hot research topic due to its wide application in clusterbased large-scale learning, federated learning, edge computing and so on. Most traditional distributed learning methods typically assume no failure or attack. However, many unexpected cases, such as communication failure and even malicious attack, may happen in real applications. Hence, Byzantine learning (BL), which refers to distributed learning with failure or attack, has recently attracted much attention. Most existing BL methods are synchronous, which are impractical in some applications due to heterogeneous or offline workers. In these cases, asynchronous BL (ABL) is usually preferred. In this paper, we propose a novel method, called buffered asynchronous stochastic gradient descent (BASGD), for ABL. To the best of our knowledge, BASGD is the first ABL method that can resist non-omniscient attacks without storing any instances on server. Furthermore, we also propose an improved variant of BASGD, called BASGD with momentum (BASGDm), by introducing momentum into BASGD. BASGDm can resist both non-omniscient and omniscient attacks. Compared with those methods which need to store instances on server, BASGD and BASGDm have a wider scope of application. Both BASGD and BASGDm are compatible with various aggregation rules. Moreover, both BASGD and BASGDm are proved to be convergent and be able to resist failure or attack. Empirical results show that our methods significantly outperform existing ABL baselines when there exists failure or attack on workers.

Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach

Byzantine-robust decentralized stochastic optimization with stochastic gradient noise-independent learning error

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

Adaptive Distributed Learning with Byzantine Robustness: A Gradient-Projection-Based Method

Efficient Byzantine-Resilient Stochastic Gradient Desce

Distributed Online Learning with Adversarial Participants in an Adversarial Environment

Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environment

Byzantine-Robust Decentralized Stochastic Optimization over Static and Time-Varying Networks

Byzantine-Robust Distributed Learning with Compression.

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation

Byzantine-robust Distributed Support Vector Machine

Resilient to byzantine attacks finite-sum optimization over networks

Distributed Sparse Learning for Stochastic Configuration Networks Via Alternating Direction Method of Multipliers

Buffered Asynchronous SGD for Byzantine Learning

High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers

Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences

Byzantine-Resilient Decentralized Collaborative Learning.