Abstract:In this paper we revisit some classic problems on classification under misspecification. In particular, we study the problem of learning halfspaces under Massart noise with rate $\eta$. In a recent work, Diakonikolas, Goulekakis, and Tzamos resolved a long-standing problem by giving the first efficient algorithm for learning to accuracy $\eta + \epsilon$ for any $\epsilon > 0$. However, their algorithm outputs a complicated hypothesis, which partitions space into $\text{poly}(d,1/\epsilon)$ regions. Here we give a much simpler algorithm and in the process resolve a number of outstanding open questions: (1) We give the first proper learner for Massart halfspaces that achieves $\eta + \epsilon$. We also give improved bounds on the sample complexity achievable by polynomial time algorithms. (2) Based on (1), we develop a blackbox knowledge distillation procedure to convert an arbitrarily complex classifier to an equally good proper classifier. (3) By leveraging a simple but overlooked connection to evolvability, we show any SQ algorithm requires super-polynomially many queries to achieve $\mathsf{OPT} + \epsilon$. Moreover we study generalized linear models where $\mathbb{E}[Y|\mathbf{X}] = \sigma(\langle \mathbf{w}^*, \mathbf{X}\rangle)$ for any odd, monotone, and Lipschitz function $\sigma$. This family includes the previously mentioned halfspace models as a special case, but is much richer and includes other fundamental models like logistic regression. We introduce a challenging new corruption model that generalizes Massart noise, and give a general algorithm for learning in this setting. Our algorithms are based on a small set of core recipes for learning to classify in the presence of misspecification. Finally we study our algorithm for learning halfspaces under Massart noise empirically and find that it exhibits some appealing fairness properties.

Non-Convex SGD Learns Halfspaces with Adversarial Label Noise

Efficient Testable Learning of General Halfspaces with Adversarial Label Noise

Provable Robustness of Adversarial Training for Learning Halfspaces with Noise

Efficiently Learning Adversarially Robust Halfspaces with Noise

Efficient PAC Learning of Halfspaces with Constant Malicious Noise Rate

Reliable Learning of Halfspaces under Gaussian Marginals

Attribute-Efficient Learning of Halfspaces with Malicious Noise: Near-Optimal Label Complexity and Noise Tolerance

Online Learning of Halfspaces with Massart Noise

Efficient Active Learning Halfspaces with Tsybakov Noise: A Non-convex Optimization Approach

Enhancing PAC Learning of Half spaces Through Robust Optimization Techniques

Agnostic Learnability of Halfspaces via Logistic Loss

Efficient Algorithms for Learning Monophonic Halfspaces in Graphs

On PAC Learning Halfspaces in Non-interactive Local Privacy Model with Public Unlabeled Data

Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to Evolvability

Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees

Generalization Bounds for Label Noise Stochastic Gradient Descent

Smoothed Analysis for Learning Concepts with Low Intrinsic Dimension

Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting

Tester-Learners for Halfspaces: Universal Algorithms

Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise

High-dimensional SGD aligns with emerging outlier eigenspaces