Abstract:Fundamental machine learning theory shows that different samples contribute unequally both in learning and testing processes. Contemporary studies on DNN imply that such sample difference is rooted on the distribution of intrinsic pattern information, namely sample regularity. Motivated by the recent discovery on network memorization and generalization, we proposed a pair of sample regularity measures for both processes with a formulation-consistent representation. Specifically, cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classiffcations of the training/testing sample within training stage, is proposed to quantize the stability in memorization-generalization process; while forgetting/mal-generalizing events, i.e., the mis-classification of previously learned or generalized sample, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. Experiments validated the effectiveness and robustness of the proposed approaches for mini-batch SGD optimization. Further applications on training/testing sample selection show the proposed measures sharing the unified computing procedure could benefit for both tasks.

The Generalization Performance of Regularized Regression Algorithms Based on Markov Sampling

Generalization Performance of Least-Square Regularized Regression Algorithm with Markov Chain Samples

The Generalization Ability of SVM Classification Based on Markov Sampling

Generalization Performance of Gaussian Kernels SVMC Based on Markov Sampling.

The Learning Performance of Support Vector Machine Classification Based on Markov Sampling

Generalization Bounds of ERM Algorithm with Markov Chain Samples

Generalization Capacity of Multi-class SVM Based on Markovian Resampling

Learning from uniformly ergodic Markov chains

M-estimator Based Support Vector Machine and Its Application

Learning performance of regularized regression with multiscale kernels based on Markov observations

Generalization Performance of Fisher Linear Discriminant Based on Markov Sampling

Learning Performance of Weighted Distributed Learning With Support Vector Machines

The Generalization Performance of ERM Algorithm with Strongly Mixing Observations

Unified Regularity Measures for Sample-wise Learning and Generalization

SVM-Boosting based on Markov resampling: Theory and algorithm

Support Vector Machines Regression with Unbounded Sampling

Generalization Bounds of Erm Algorithm with V-Geometrically Ergodic Markov Chains

K-Times Markov Sampling for SVMC

Generalization bounds for nonparametric regression with $β-$mixing samples

Generalization Performance Analysis of M-SVMs

Learning performance of Tikhonov regularization algorithm with geometrically beta-mixing observations☆