Abstract:Much of learning theory is concerned with the design and analysis of probably approximately correct (PAC) learners. The closely related transductive model of learning has recently seen more scrutiny, with its learners often used as precursors to PAC learners. Our goal in this work is to understand and quantify the exact relationship between these two models. First, we observe that modest extensions of existing results show the models to be essentially equivalent for realizable learning for most natural loss functions, up to low order terms in the error and sample complexity. The situation for agnostic learning appears less straightforward, with sample complexities potentially separated by a $\frac{1}{\epsilon}$ factor. This is therefore where our main contributions lie. Our results are two-fold: 1. For agnostic learning with bounded losses (including, for example, multiclass classification), we show that PAC learning reduces to transductive learning at the cost of low-order terms in the error and sample complexity via an adaptation of the reduction of <a class="link-https" data-arxiv-id="2304.09167" href="https://arxiv.org/abs/2304.09167">arXiv:2304.09167</a> to the agnostic setting. 2. For agnostic binary classification, we show the converse: transductive learning is essentially no more difficult than PAC learning. Together with our first result this implies that the PAC and transductive models are essentially equivalent for agnostic binary classification. This is our most technical result, and involves two steps: A symmetrization argument on the agnostic one-inclusion graph (OIG) of <a class="link-https" data-arxiv-id="2309.13692" href="https://arxiv.org/abs/2309.13692">arXiv:2309.13692</a> to derive the worst-case agnostic transductive instance, and expressing the error of the agnostic OIG algorithm for this instance in terms of the empirical Rademacher complexity of the class. We leave as an intriguing open question whether our second result can be extended beyond binary classification to show the transductive and PAC models equivalent more broadly.

Permutational Rademacher Complexity: a New Complexity Measure for Transductive Learning

Researches on Rademacher Complexities in Statistical Learning Theory：A Survey

Sharp Generalization of Transductive Learning: A Transductive Local Rademacher Complexity Approach

A Measure of the Complexity of Neural Representations based on Partial Information Decomposition

Local Rademacher Complexity for Multi-label Learning.

Is Transductive Learning Equivalent to PAC Learning?

Complexity Measures and Concept Learning

Approximate Inference via Weighted Rademacher Complexity

Permutation Decision Trees

Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization

Covering Number As a Complexity Measure for POMDP Planning and Learning.

Information-Theoretic Generalization Bounds for Transductive Learning and its Applications

Simpler and Better Cardinality Estimators for HyperLogLog and PCSA

Exact Count of Boundary Pieces of ReLU Classifiers: Towards the Proper Complexity Measure for Classification

Rademacher Complexity for Adversarially Robust Generalization.

Level Up: Private Non-Interactive Decision Tree Evaluation using Levelled Homomorphic Encryption

A Novel Measure of Attribute Significance with Complexity Weight

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Dropout Rademacher Complexity of Deep Neural Networks.

Reasoning and Predicting POMDP Planning Complexity Via Covering Numbers

Oracle Complexity and Nontransitivity in Pattern Recognition