Abstract:Learning problems form an important category of computational tasks that generalizes many of the computations researchers apply to large real-life data sets. We ask, What concept classes can be learned privately, namely, by an algorithm whose output does not depend too heavily on any one input or specific training example? More precisely, we investigate learning algorithms that satisfy differential privacy, a notion that provides strong confidentiality guarantees in contexts where aggregate information is released about a database containing sensitive information about individuals. Our goal is a broad understanding of the resources required for private learning in terms of samples, computation time, and interaction. We demonstrate that, ignoring computational constraints, it is possible to privately agnostically learn any concept class using a sample size approximately logarithmic in the cardinality of the concept class. Therefore, almost anything learnable is learnable privately: specifically, if a concept class is learnable by a (nonprivate) algorithm with polynomial sample complexity and output size, then it can be learned privately using a polynomial number of samples. We also present a computationally efficient private probabilistically approximately correct learner for the class of parity functions. This result dispels the similarity between learning with noise and private learning (both must be robust to small changes in inputs), since parity is thought to be very hard to learn given random classification noise. Local (or randomized response) algorithms are a practical class of private algorithms that have received extensive investigation. We provide a precise characterization of local private learning algorithms. We show that a concept class is learnable by a local algorithm if and only if it is learnable in the statistical query (SQ) model. Therefore, for local private learning algorithms, the similarity to learning with noise is stronger: local learning is equivalent to SQ learning, and SQ algorithms include most known noise-tolerant learning algorithms. Finally, we present a separation between the power of interactive and noninteractive local learning algorithms. Because of the equivalence to SQ learning, this result also separates adaptive and nonadaptive SQ learning.

Private Learning Implies Online Learning: An Efficient Reduction

A Computational Separation between Private Learning and Online Learning

Privacy Risks of Social Interaction Structure: Network Learning in Quadratic Games

Do you pay for Privacy in Online learning?

The Limits of Differential Privacy in Online Learning

Differentially Private Nash Equilibrium Seeking in Quadratic Network Games

Private Online Learning via Lazy Algorithms

Oracle-Efficient Differentially Private Learning with Public Data

Near-Optimal Algorithms for Differentially Private Online Learning in a Stochastic Environment

Private PAC Learning May be Harder than Online Learning

Offline Reinforcement Learning with Differential Privacy

Differentially Private Fair Learning

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

What Can We Learn Privately?

Differentially Private Nonparametric Regression Under a Growth Condition

Preserving Expert-Level Privacy in Offline Reinforcement Learning

Differentially private Nash equilibrium seeking for networked aggregative games

Differentially Private Reinforcement Learning with Self-Play

Evaluating Differentially Private Machine Learning in Practice

Online Sensitivity Optimization in Differentially Private Learning

Learning to be adversarially robust and differentially private