Abstract:In many real-world applications, due to recent developments in the privacy landscape, training data may be aggregated to preserve the privacy of sensitive training labels. In the learning from label proportions (LLP) framework, the dataset is partitioned into bags of feature-vectors which are available only with the sum of the labels per bag. A further restriction, which we call learning from bag aggregates (LBA) is where instead of individual feature-vectors, only the (possibly weighted) sum of the feature-vectors per bag is available. We study whether such aggregation techniques can provide privacy guarantees under the notion of label differential privacy (label-DP) previously studied in for e.g. [Chaudhuri-Hsu'11, Ghazi et al.'21, Esfandiari et al.'22]. It is easily seen that naive LBA and LLP do not provide label-DP. Our main result however, shows that weighted LBA using iid Gaussian weights with $m$ randomly sampled disjoint $k$-sized bags is in fact $(\varepsilon, \delta)$-label-DP for any $\varepsilon > 0$ with $\delta \approx \exp(-\Omega(\sqrt{k}))$ assuming a lower bound on the linear-mse regression loss. Further, the $\ell_2^2$-regressor which minimizes the loss on the aggregated dataset has a loss within $\left(1 + o(1)\right)$-factor of the optimum on the original dataset w.p. $\approx 1 - exp(-\Omega(m))$. We emphasize that no additive label noise is required. The analogous weighted-LLP does not however admit label-DP. Nevertheless, we show that if additive $N(0, 1)$ noise can be added to any constant fraction of the instance labels, then the noisy weighted-LLP admits similar label-DP guarantees without assumptions on the dataset, while preserving the utility of Lipschitz-bounded neural mse-regression tasks. Our work is the first to demonstrate that label-DP can be achieved by randomly weighted aggregation for regression tasks, using no or little additive noise.

Aggregating Data for Optimal and Private Learning

Weak to Strong Learning from Aggregate Labels

Learning from Aggregate responses: Instance Level versus Bag Level Loss Functions

Multiple-Instance Learning from Pairwise Comparison Bags

Learning from Aggregated Data: Curated Bags versus Random Bags

Learning from Label Proportions and Covariate-shifted Instances

Label Differential Privacy via Aggregation

OT-LLP: Optimal Transport for Learning from Label Proportions

Regularization and Optimal Multiclass Learning

Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation

Privacy-Preserving Multiparty Learning For Logistic Regression

MixBag: Bag-Level Data Augmentation for Learning from Label Proportions

Learning from Multi-Dimensional Partial Labels.

Fast learning from label proportions with small bags

Theoretical Proportion Label Perturbation for Learning from Label Proportions in Large Bags

PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses

A bag-to-class divergence approach to multiple-instance learning

Instance- and bag-level manifold regularization for aggregate outputs classification.

Learning from Label Proportions with Instance-wise Consistency

Learning with Sets in Multiple Instance Regression Applied to Remote Sensing

Two-stage Training for Learning from Label Proportions