Abstract:As one of the efficient approaches to deal with big data, divide-and-conquer distributed algorithms, such as the distributed kernel regression, bootstrap, structured perception training algorithms, and so on, are proposed and broadly used in learning systems. Some learning theories have been built to analyze the feasibility, approximation, and convergence bounds of these distributed learning algorithms. However, less work has been studied on the stability of these distributed learning algorithms. In this paper, we discuss the generalization bounds of distributed learning algorithms from the view of algorithmic stability. First, we introduce a definition of uniform distributed stability for distributed algorithms and study the distributed algorithms' generalization risk bounds. Then, we analyze the stability properties and generalization risk bounds of a kind of regularization-based distributed algorithms. Two generalization distributed risks obtained show that the generalization distributed risk bounds for the difference between their generalization distributed and empirical distributed/leave-one-computer-out risks are closely related to the size of samples n and the amount of working computers m mathcal O(m/n(1/2)). Furthermore, the results in this paper indicate that, for a good generalization regularized distributed kernel algorithm, the regularization parameter lambda should be adjusted with the change of the term m/n(1/2). These theoretic discoveries provide the useful guidance when deploying the distributed algorithms on practical big data platforms. We explore our theoretic analyses through two simulation experiments. Finally, we discuss some problems about the sufficient amount of working computers, nonequivalence, and generalization for distributed learning. We show that the rules for the computation on one single computer may not always hold for distributed learning.

Stability Analysis and Learning Bounds for Transductive Regression Algorithms

Algorithmic Stability Unleashed: Generalization Bounds with Unbounded Losses

Generalization Error Bounds for Optimization Algorithms Via Stability

Stability revisited: new generalisation bounds for the Leave-one-Out

Stability-Based Generalization Analysis of Distributed Learning Algorithms for Big Data

Data-Dependent Stability Analysis of Adversarial Training

Stability of decision trees and logistic regression

Stability Analysis and Generalization Bounds of Adversarial Training

Boosting the Confidence of Generalization for L2-Stable Randomized Learning Algorithms

Algorithmic stability and hypothesis complexity

Information-Theoretic Generalization Bounds for Transductive Learning and its Applications

$L_2$-Uniform Stability of Randomized Learning Algorithms: Sharper Generalization Bounds and Confidence Boosting

Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent

Strong and Weak Stability of Randomized Learning Algorithms

GENERALIZATION BOUNDS OF REGULARIZATION ALGORITHMS DERIVED SIMULTANEOUSLY THROUGH HYPOTHESIS SPACE COMPLEXITY, ALGORITHMIC STABILITY AND DATA QUALITY

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent

Fine-Grained Analysis of Stability and Generalization for Modern Meta Learning Algorithms

Exponential Generalization Bounds with Near-Optimal Rates for $L_q$-Stable Algorithms.

Stability and Generalization of the Learning Algorithm (Ⅱ): the Framework of the Bounded-Difference Stability

Can Stability be Detrimental? Better Generalization through Gradient Descent Instabilities