Abstract:In this paper we revisit some classic problems on classification under misspecification. In particular, we study the problem of learning halfspaces under Massart noise with rate $\eta$. In a recent work, Diakonikolas, Goulekakis, and Tzamos resolved a long-standing problem by giving the first efficient algorithm for learning to accuracy $\eta + \epsilon$ for any $\epsilon > 0$. However, their algorithm outputs a complicated hypothesis, which partitions space into $\text{poly}(d,1/\epsilon)$ regions. Here we give a much simpler algorithm and in the process resolve a number of outstanding open questions: (1) We give the first proper learner for Massart halfspaces that achieves $\eta + \epsilon$. We also give improved bounds on the sample complexity achievable by polynomial time algorithms. (2) Based on (1), we develop a blackbox knowledge distillation procedure to convert an arbitrarily complex classifier to an equally good proper classifier. (3) By leveraging a simple but overlooked connection to evolvability, we show any SQ algorithm requires super-polynomially many queries to achieve $\mathsf{OPT} + \epsilon$. Moreover we study generalized linear models where $\mathbb{E}[Y|\mathbf{X}] = \sigma(\langle \mathbf{w}^*, \mathbf{X}\rangle)$ for any odd, monotone, and Lipschitz function $\sigma$. This family includes the previously mentioned halfspace models as a special case, but is much richer and includes other fundamental models like logistic regression. We introduce a challenging new corruption model that generalizes Massart noise, and give a general algorithm for learning in this setting. Our algorithms are based on a small set of core recipes for learning to classify in the presence of misspecification. Finally we study our algorithm for learning halfspaces under Massart noise empirically and find that it exhibits some appealing fairness properties.

What problem does this paper attempt to address?

This paper attempts to solve the problem of classification in the presence of model misspecification. Specifically, the paper focuses on the problem of learning half - spaces when the labels are contaminated by Massart noise. Massart noise means that the label of each sample is flipped with a certain probability, which can depend on the sample itself but does not exceed a fixed threshold η (η < 1/2). The main contributions of the paper are to provide several new algorithms and techniques that can efficiently learn half - spaces under such noise conditions and have good theoretical guarantees. The following are the specific problems that the paper attempts to solve: 1. **Providing correct learning algorithms**: - The paper proposes the first proper algorithm for learning half - spaces under Massart noise, which can achieve an error of η + ϵ, where ϵ > 0. Unlike previous methods, this algorithm outputs a simple half - space instead of dividing the space into a polynomial number of regions. - The paper also improves the bounds on sample complexity for polynomial - time algorithms. 2. **Knowledge distillation**: - Based on the above correct learning algorithm, the paper develops a black - box knowledge distillation method, which can transform any complex classifier into a proper half - space with the same prediction accuracy. 3. **Lower bounds on optimal accuracy**: - The paper gives for the first time the lower bounds on achieving optimal accuracy (OPT + ϵ) under Massart noise. In particular, the paper proves a super - polynomial Statistical Query (SQ) lower bound, which indicates that in some cases, even the optimal algorithm requires a super - polynomial number of queries to achieve the target accuracy. 4. **Extension to generalized linear models**: - The paper further studies Generalized Linear Models (GLMs), where the conditional expectation $ E[Y|X]=\sigma(\langle w^*, X \rangle) $, and the function σ is an odd function, monotonic and Lipschitz continuous. The paper introduces a new noise model that generalizes Massart noise and gives a learning algorithm in this setting. 5. **Experimental verification**: - The paper experimentally verifies the proposed algorithms on synthetic data and real - world datasets (such as the UCI Adult dataset) and finds that these algorithms not only perform well in overall accuracy but also can avoid adverse biases in different groups. Through these contributions, the paper significantly advances the research on classification in the presence of model misspecification both theoretically and practically.

Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to Evolvability

Online Learning of Halfspaces with Massart Noise

Non-Adaptive Stochastic Score Classification and Explainable Halfspace Evaluation

Attribute-Efficient Learning of Halfspaces with Malicious Noise: Near-Optimal Label Complexity and Noise Tolerance

Efficient Testable Learning of General Halfspaces with Adversarial Label Noise

Non-Convex SGD Learns Halfspaces with Adversarial Label Noise

Tester-Learners for Halfspaces: Universal Algorithms

Reliable Learning of Halfspaces under Gaussian Marginals

Provable Robustness of Adversarial Training for Learning Halfspaces with Noise

Efficient Active Learning Halfspaces with Tsybakov Noise: A Non-convex Optimization Approach

Agnostic Learnability of Halfspaces via Logistic Loss

On the Model-Misspecification in Reinforcement Learning

Prediction, Learning, Uniform Convergence, and Scale-sensitive Dimensions

Efficient PAC Learning of Halfspaces with Constant Malicious Noise Rate

Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning

Semi-supervised Regression Analysis with Model Misspecification and High-dimensional Data

Misspecified $Q$-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Error

Contrastive Moments: Unsupervised Halfspace Learning in Polynomial Time

Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting

Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models

Belief Convergence under Misspecified Learning: A Martingale Approach