Abstract:Byzantine-robust distributed learning (BRDL), in which computing devices are likely to behave abnormally due to accidental failures or malicious attacks, has recently become a hot research topic. However, even in the independent and identically distributed (i.i.d.) case, existing BRDL methods will suffer a significant drop on model accuracy due to the large variance of stochastic gradients. Increasing batch sizes is a simple yet effective way to reduce the variance. However, when the total number of gradient computation is fixed, a too-large batch size will lead to a too-small iteration number (update number), which may also degrade the model accuracy. In view of this challenge, we mainly study the effect of batch size when the total number of gradient computation is fixed in this work. In particular, we show that when the total number of gradient computation is fixed, the optimal batch size corresponding to the tightest theoretical upper bound in BRDL increases with the fraction of Byzantine workers. Therefore, compared to the case without attacks, a larger batch size is preferred when under Byzantine attacks. Motivated by the theoretical finding, we propose a novel method called Byzantine-robust stochastic gradient descent with normalized momentum (ByzSGDnm) in order to further increase model accuracy in BRDL. We theoretically prove the convergence of ByzSGDnm for general non-convex cases under Byzantine attacks. Empirical results show that when under Byzantine attacks, compared to the cases of small batch sizes, setting a relatively large batch size can significantly increase the model accuracy, which is consistent with our theoretical results. Moreover, ByzSGDnm can achieve higher model accuracy than existing BRDL methods when under deliberately crafted attacks. In addition, we empirically show that increasing batch sizes has the bonus of training acceleration.

BASGD: Buffered Asynchronous SGD for Byzantine Learning

Buffered Asynchronous SGD for Byzantine Learning

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Efficient Byzantine-Resilient Stochastic Gradient Desce

On the Optimal Batch Size for Byzantine-Robust Distributed Learning

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Adaptive Distributed Learning with Byzantine Robustness: A Gradient-Projection-Based Method

Justinian's GAAvernor: Robust Distributed Learning with Gradient Aggregation Agent.

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Byzantine-resilient Decentralized Stochastic Gradient Descent

On the Effect of Batch Size in Byzantine-Robust Distributed Learning

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

The Vital Role of Gradient Clipping in Byzantine-Resilient Distributed Learning

High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks

Generalization Error Matters in Decentralized Learning Under Byzantine Attacks

Mitigating Model Poisoning Attacks on Distributed Learning with Heterogeneous Data

Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Data

Asynchronous Byzantine Federated Learning

Byzantine-Resilient Non-Convex Stochastic Gradient Descent

BEV-SGD: Best Effort Voting SGD for Analog Aggregation Based Federated Learning against Byzantine Attackers