Abstract:As one of the efficient approaches to deal with big data, divide-and-conquer distributed algorithms, such as the distributed kernel regression, bootstrap, structured perception training algorithms, and so on, are proposed and broadly used in learning systems. Some learning theories have been built to analyze the feasibility, approximation, and convergence bounds of these distributed learning algorithms. However, less work has been studied on the stability of these distributed learning algorithms. In this paper, we discuss the generalization bounds of distributed learning algorithms from the view of algorithmic stability. First, we introduce a definition of uniform distributed stability for distributed algorithms and study the distributed algorithms' generalization risk bounds. Then, we analyze the stability properties and generalization risk bounds of a kind of regularization-based distributed algorithms. Two generalization distributed risks obtained show that the generalization distributed risk bounds for the difference between their generalization distributed and empirical distributed/leave-one-computer-out risks are closely related to the size of samples n and the amount of working computers m mathcal O(m/n(1/2)). Furthermore, the results in this paper indicate that, for a good generalization regularized distributed kernel algorithm, the regularization parameter lambda should be adjusted with the change of the term m/n(1/2). These theoretic discoveries provide the useful guidance when deploying the distributed algorithms on practical big data platforms. We explore our theoretic analyses through two simulation experiments. Finally, we discuss some problems about the sufficient amount of working computers, nonequivalence, and generalization for distributed learning. We show that the rules for the computation on one single computer may not always hold for distributed learning.

Stability and Generalization of the Learning Algorithm (Ⅱ): the Framework of the Bounded-Difference Stability

A new approach to almost-sure asymptotic stability of stochastic systems of higher dimension

Stability-Based Generalization Analysis of Distributed Learning Algorithms for Big Data

$L_2$-Uniform Stability of Randomized Learning Algorithms: Sharper Generalization Bounds and Confidence Boosting

Algorithmic Stability Unleashed: Generalization Bounds with Unbounded Losses

Stability Analysis and Learning Bounds for Transductive Regression Algorithms

Boosting the Confidence of Generalization for L2-Stable Randomized Learning Algorithms

GENERALIZATION BOUNDS OF REGULARIZATION ALGORITHMS DERIVED SIMULTANEOUSLY THROUGH HYPOTHESIS SPACE COMPLEXITY, ALGORITHMIC STABILITY AND DATA QUALITY

A Stability Principle for Learning under Non-Stationarity

Strong and Weak Stability of Randomized Learning Algorithms

Generalization Error Bounds for Optimization Algorithms Via Stability

Approximation Stability And Boosting

Stability revisited: new generalisation bounds for the Leave-one-Out

Kernel Stability for Model Selection in Kernel-Based Algorithms.

Algorithmic stability and hypothesis complexity

Stability for the training of deep neural networks and other classifiers

Stability Analysis of Learning Algorithms for Blind Source Separation

The Bayesian Stability Zoo

Stability of decision trees and logistic regression

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent