Abstract:With powerful parallel computing GPUs and massive user data, neural-network-based deep learning can well exert its strong power in problem modeling and solving, and has archived great success in many applications such as image classification, speech recognition and machine translation etc. While deep learning has been increasingly popular, the problem of privacy leakage becomes more and more urgent. Given the fact that the training data may contain highly sensitive information, e.g., personal medical records, directly sharing them among the users (i.e., participants) or centrally storing them in one single location may pose a considerable threat to user privacy. In this paper, we present a practical privacy-preserving collaborative deep learning system that allows users to cooperatively build a collective deep learning model with data of all participants, without direct data sharing and central data storage. In our system, each participant trains a local model with their own data and only shares model parameters with the others. To further avoid potential privacy leakage from sharing model parameters, we use functional mechanism to perturb the objective function of the neural network in the training process to achieve $\epsilon $ -differential privacy. In particular, for the first time, we consider the existence of unreliable participants , i.e., the participants with low-quality data, and propose a solution to reduce the impact of these participants while protecting their privacy. We evaluate the performance of our system on two well-known real-world datasets for regression and classification tasks. The results demonstrate that the proposed system is robust against unreliable participants, and achieves high accuracy close to the model trained in a traditional centralized manner while ensuring rigorous privacy protection.

Max-Min-Degree Neural Network for Centralized-Decentralized Collaborative Computing.

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

A Graph Neural Network Based Decentralized Learning Scheme

K-Degree Layer-Wise Network for Geo-Distributed Computing Between Cloud and IoT

Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence

Decentralized Deep Learning using Momentum-Accelerated Consensus

Collaborative Deep Learning in Fixed Topology Networks

Energy-efficient Decentralized Learning via Graph Sparsification

Collaborative Deep Learning Across Multiple Data Centers

Decentralized Learning Framework for Hierarchical Wireless Networks: A Tree Neural Network Approach

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

Dominating Set Model Aggregation for communication-efficient decentralized deep learning

Learning to Collaborate in Decentralized Learning of Personalized Models

MinMax Networks

Nonlinear Collaborative Scheme for Deep Neural Networks.

A Communication-Efficient Model of Sparse Neural Network for Distributed Intelligence

Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Rate

Automatic Organization of Neural Modules for Enhanced Collaboration in Neural Networks

Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge

Network Fault-tolerant and Byzantine-resilient Social Learning via Collaborative Hierarchical Non-Bayesian Learning

A collaborative neurodynamic approach with two-timescale projection neural networks designed via majorization-minimization for global optimization and distributed global optimization