Abstract:Learning problems form an important category of computational tasks that generalizes many of the computations researchers apply to large real-life data sets. We ask, What concept classes can be learned privately, namely, by an algorithm whose output does not depend too heavily on any one input or specific training example? More precisely, we investigate learning algorithms that satisfy differential privacy, a notion that provides strong confidentiality guarantees in contexts where aggregate information is released about a database containing sensitive information about individuals. Our goal is a broad understanding of the resources required for private learning in terms of samples, computation time, and interaction. We demonstrate that, ignoring computational constraints, it is possible to privately agnostically learn any concept class using a sample size approximately logarithmic in the cardinality of the concept class. Therefore, almost anything learnable is learnable privately: specifically, if a concept class is learnable by a (nonprivate) algorithm with polynomial sample complexity and output size, then it can be learned privately using a polynomial number of samples. We also present a computationally efficient private probabilistically approximately correct learner for the class of parity functions. This result dispels the similarity between learning with noise and private learning (both must be robust to small changes in inputs), since parity is thought to be very hard to learn given random classification noise. Local (or randomized response) algorithms are a practical class of private algorithms that have received extensive investigation. We provide a precise characterization of local private learning algorithms. We show that a concept class is learnable by a local algorithm if and only if it is learnable in the statistical query (SQ) model. Therefore, for local private learning algorithms, the similarity to learning with noise is stronger: local learning is equivalent to SQ learning, and SQ algorithms include most known noise-tolerant learning algorithms. Finally, we present a separation between the power of interactive and noninteractive local learning algorithms. Because of the equivalence to SQ learning, this result also separates adaptive and nonadaptive SQ learning.

On The Robustness Of Learning Parity With Noise

Cryptography with Auxiliary Input and Trapdoor from Constant-Noise LPN.

On the Hardness of Learning Parity with Noise over Rings.

On Learning Parities with Dependent Noise

Algorithms for Sparse LPN and LSPN Against Low-noise

Collision Resistant Hashing From Sub-Exponential Learning Parity With Noise

The Learning Stabilizers with Noise problem

Lossy Cryptography from Code-Based Assumptions

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks

On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning

Reduction from sparse LPN to LPN, Dual Attack 3.0

On the Vulnerability of Fairness Constrained Learning to Malicious Noise

Sample Complexity of Robust Learning against Evasion Attacks

Learning Constant-Depth Circuits in Malicious Noise Models

It begins with a boundary: A geometric view on probabilistically robust learning

What Can We Learn Privately?

Limitations of the decoding-to-LPN reduction via code smoothing

Certifiably Robust Encoding Schemes

RoLNiP: Robust Learning Using Noisy Pairwise Comparisons

Intractability of Learning the Discrete Logarithm with Gradient-Based Methods

The Privacy Power of Correlated Noise in Decentralized Learning