Abstract:In many real-world applications, due to recent developments in the privacy landscape, training data may be aggregated to preserve the privacy of sensitive training labels. In the learning from label proportions (LLP) framework, the dataset is partitioned into bags of feature-vectors which are available only with the sum of the labels per bag. A further restriction, which we call learning from bag aggregates (LBA) is where instead of individual feature-vectors, only the (possibly weighted) sum of the feature-vectors per bag is available. We study whether such aggregation techniques can provide privacy guarantees under the notion of label differential privacy (label-DP) previously studied in for e.g. [Chaudhuri-Hsu'11, Ghazi et al.'21, Esfandiari et al.'22]. It is easily seen that naive LBA and LLP do not provide label-DP. Our main result however, shows that weighted LBA using iid Gaussian weights with $m$ randomly sampled disjoint $k$-sized bags is in fact $(\varepsilon, \delta)$-label-DP for any $\varepsilon > 0$ with $\delta \approx \exp(-\Omega(\sqrt{k}))$ assuming a lower bound on the linear-mse regression loss. Further, the $\ell_2^2$-regressor which minimizes the loss on the aggregated dataset has a loss within $\left(1 + o(1)\right)$-factor of the optimum on the original dataset w.p. $\approx 1 - exp(-\Omega(m))$. We emphasize that no additive label noise is required. The analogous weighted-LLP does not however admit label-DP. Nevertheless, we show that if additive $N(0, 1)$ noise can be added to any constant fraction of the instance labels, then the noisy weighted-LLP admits similar label-DP guarantees without assumptions on the dataset, while preserving the utility of Lipschitz-bounded neural mse-regression tasks. Our work is the first to demonstrate that label-DP can be achieved by randomly weighted aggregation for regression tasks, using no or little additive noise.

Leveraged Weighted Loss for Partial Label Learning (supplementary Material)

Partial-Label Regression

Learning from Multi-Dimensional Partial Labels.

Partial Multi-label Learning with Label and Feature Collaboration

On the consistency of multi-label learning

Loss factorization, weakly supervised learning and label noise robustness

Leveraged Asymmetric Loss with Disambiguation for Multi-label Recognition with One-Positive Annotations

Multi-Label Learning with Stronger Consistency Guarantees

Adversary-Aware Partial label learning with Label distillation

Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical

Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions

Robust Long-Tailed Learning Via Label-Aware Bounded CVaR

Learning from Label Proportions with Instance-wise Consistency

Student Loss: Towards the Probability Assumption in Inaccurate Supervision

Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity

Class-Imbalanced Complementary-Label Learning via Weighted Loss

Label Differential Privacy via Aggregation

Adversarial Partial Multi-Label Learning

Learning from Label Proportions with Consistency Regularization

Partial Label Learning with Unlabeled Data.