Abstract:Machine learning has begun to play a central role in many applications. A multitude of these applications typically also involve datasets that are distributed across multiple computing devices/machines due to either design constraints (e.g., multiagent systems) or computational/privacy reasons (e.g., learning on smartphone data). Such applications often require the learning tasks to be carried out in a decentralized fashion, in which there is no central server that is directly connected to all nodes. In real-world decentralized settings, nodes are prone to undetected failures due to malfunctioning equipment, cyberattacks, etc., which are likely to crash non-robust learning algorithms. The focus of this paper is on robustification of decentralized learning in the presence of nodes that have undergone Byzantine failures. The Byzantine failure model allows faulty nodes to arbitrarily deviate from their intended behaviors, thereby ensuring designs of the most robust of algorithms. But the study of Byzantine resilience within decentralized learning, in contrast to distributed learning, is still in its infancy. In particular, existing Byzantine-resilient decentralized learning methods either do not scale well to large-scale machine learning models, or they lack statistical convergence guarantees that help characterize their generalization errors. In this paper, a scalable, Byzantine-resilient decentralized machine learning framework termed Byzantine-resilient decentralized gradient descent (BRIDGE) is introduced. Algorithmic and statistical convergence guarantees for one variant of BRIDGE are also provided in the paper for both strongly convex problems and a class of nonconvex problems. In addition, large-scale decentralized learning experiments are used to establish that the BRIDGE framework is scalable and it delivers competitive results for Byzantine-resilient convex and nonconvex learning.

Byzantine-resilient decentralized stochastic gradient descent

Byzantine-resilient Decentralized Stochastic Gradient Descent

Byzantine-resilient Decentralized Network Learning

Efficient Byzantine-Resilient Stochastic Gradient Desce

Generalization Error Matters in Decentralized Learning Under Byzantine Attacks

Byzantine-robust decentralized stochastic optimization with stochastic gradient noise-independent learning error

On the Generalization Error of Byzantine-Resilient Decentralized Learning.

Byzantine-Resilient Decentralized Collaborative Learning.

Byzantine-Resilient Non-Convex Stochastic Gradient Descent

A Graph Neural Network Based Decentralized Learning Scheme

BRIDGE: Byzantine-resilient Decentralized Gradient Descent

Adaptive Distributed Learning with Byzantine Robustness: A Gradient-Projection-Based Method

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Towards byzantine-resilient learning in decentralized systems

Robust Decentralized Stochastic Gradient Descent over Unstable Networks.

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Practical Differentially Private and Byzantine-resilient Federated Learning

Resilient to byzantine attacks finite-sum optimization over networks

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy