Abstract:The goal of a learning algorithm is to receive a training data set as input and provide a hypothesis that can generalize to all possible data points from a domain set. The hypothesis is chosen from hypothesis classes with potentially different complexities. Linear regression modeling is an important category of learning algorithms. The practical uncertainty of the target samples affects the generalization performance of the learned model. Failing to choose a proper model or hypothesis class can lead to serious issues such as underfitting or overfitting. These issues have been addressed by alternating cost functions or by utilizing cross-validation methods. These approaches can introduce new hyperparameters with their own new challenges and uncertainties or increase the computational complexity of the learning algorithm. On the other hand, the theory of probably approximately correct (PAC) aims at defining learnability based on probabilistic settings. Despite its theoretical value, PAC does not address practical learning issues on many occasions. This work is inspired by the foundation of PAC and is motivated by the existing regression learning issues. The proposed approach, denoted by epsilon-Confidence Approximately Correct (epsilon CoAC), utilizes Kullback Leibler divergence (relative entropy) and proposes a new related typical set in the set of hyperparameters to tackle the learnability issue. Moreover, it enables the learner to compare hypothesis classes of different complexity orders and choose among them the optimum with the minimum epsilon in the epsilon CoAC framework. Not only the epsilon CoAC learnability overcomes the issues of overfitting and underfitting, but it also shows advantages and superiority over the well known cross-validation method in the sense of time consumption as well as in the sense of accuracy.

Learning Coefficients in Semi-Regular Models

Real Log Canonical Thresholds at Non-singular Points

Structure Learning in Inverse Ising Problems Using ℓ_2-Regularized Linear Estimator

Learning Theory Estimates for Coefficient-Based Regularized Regression

Statistical Learning Theory of Quasi-Regular Cases

Coefficient-based regularized distribution regression

Estimating the Local Learning Coefficient at Scale

Analysis of Regularized Learning for Linear-functional Data in Banach Spaces

Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Models

Learnability, Sample Complexity, and Hypothesis Class Complexity for Regression Models

Learning under Singularity: An Information Criterion improving WBIC and sBIC

On the maximum likelihood estimation in general log-linear models

Consideration on the learning efficiency of multiple-layered neural networks with linear units

Information theoretic limits of learning a sparse rule

High-dimensional varying index coefficient quantile regression model

Regularized EM Algorithms: A Unified Framework and Statistical Guarantees

Optimal Rates for Coefficient-Based Regularized Regression

Learning rates for regularized least squares ranking algorithm

Classification of real hyperplane singularities by real log canonical thresholds

A Unified Theory of Exact Inference and Learning in Exponential Family Latent Variable Models

A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection