Abstract:Secure multi-party computation-based machine learning, referred to as MPL, has become an important technology to utilize data from multiple parties with privacy preservation. While MPL provides rigorous security guarantees for the computation process, the models trained by MPL are still vulnerable to attacks that solely depend on access to the models. Differential privacy could help to defend against such attacks. However, the accuracy loss brought by differential privacy and the huge communication overhead of secure multi-party computation protocols make it highly challenging to balance the 3-way trade-off between privacy, efficiency, and accuracy. In this paper, we are motivated to resolve the above issue by proposing a solution, referred to as PEA (Private, Efficient, Accurate), which consists of a secure DPSGD protocol and two optimization methods. First, we propose a secure DPSGD protocol to enforce DPSGD in secret sharing-based MPL frameworks. Second, to reduce the accuracy loss led by differential privacy noise and the huge communication overhead of MPL, we propose two optimization methods for the training process of MPL: (1) the data-independent feature extraction method, which aims to simplify the trained model structure; (2) the local data-based global model initialization method, which aims to speed up the convergence of the model training. We implement PEA in two open-source MPL frameworks: TF-Encrypted and Queqiao. The experimental results on various datasets demonstrate the efficiency and effectiveness of PEA. E.g. when ${\epsilon}$ = 2, we can train a differentially private classification model with an accuracy of 88% for CIFAR-10 within 7 minutes under the LAN setting. This result significantly outperforms the one from CryptGPU, one SOTA MPL framework: it costs more than 16 hours to train a non-private deep neural network model on CIFAR-10 with the same accuracy.

Personalized and Private Peer-to-Peer Machine Learning

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

P4: Towards private, personalized, and Peer-to-Peer learning

Efficient Node Selection in Private Personalized Decentralized Learning

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

PD-ML-Lite: Private Distributed Machine Learning from Lighweight Cryptography

CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning

Muffliato: Peer-to-Peer Privacy Amplification for Decentralized Optimization and Averaging

The Cost of Privacy in Asynchronous Differentially-Private Machine Learning

Hiding in the Crowd: A Massively Distributed Algorithm for Private Averaging with Malicious Adversaries

MAPL: Model Agnostic Peer-to-peer Learning

Fairness-Driven Private Collaborative Machine Learning

Protection Against Reconstruction and Its Applications in Private Federated Learning

SecureML: A System for Scalable Privacy-Preserving Machine Learning

SPEED: Secure, PrivatE, and Efficient Deep learning

Efficient Deep Learning on Multi-Source Private Data

Collaborative Mean Estimation over Intermittently Connected Networks with Peer-To-Peer Privacy

Private, Efficient, and Accurate: Protecting Models Trained by Multi-party Learning with Differential Privacy

Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning

Decentralized Stochastic Optimization with Inherent Privacy Protection

Echo of Neighbors: Privacy Amplification for Personalized Private Federated Learning with Shuffle Model