Abstract:Deep learning (DL) models are increasingly built on federated edge participants holding local data. To enable insight extractions without the risk of information leakage, DL training is usually combined with differential privacy (DP). The core theme is to tradeoff learning accuracy by adding statistically calibrated noises, particularly to local gradients of edge learners, during model training. However, this privacy guarantee unfortunately degrades model accuracy due to edge learners' local noises, and the global noise aggregated at the central server. Existing DP frameworks for edge focus on local noise calibration via gradient clipping techniques, overlooking the heterogeneity and dynamic changes of local gradients, and their aggregated impact on accuracy. In this article, we present a systematical analysis that unveils the influential factors capable of mitigating local and aggregated noises, and design PrivateDL to leverage these factors in noise calibration so as to improve model accuracy while fulfilling privacy guarantee. PrivateDL features on: (i) sampling-based sensitivity estimation for local noise calibration and (ii) combining large batch sizes and critical data identification in global training. We implement PrivateDL on the popular Laplace/Gaussian DP mechanisms and demonstrate its effectiveness using Intel BigDL workloads, i.e., considerably improving model accuracy by up to 5X when comparing against existing DP frameworks.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the accuracy of the model while ensuring data privacy when conducting deep - learning training on edge devices. Specifically, the paper focuses on how to reduce the problem of model performance degradation caused by adding noise in the process of protecting data privacy through differential privacy technology under the federated learning framework. ### Paper Background With the development of edge computing, more and more deep - learning (DL) models begin to be trained using local private data on multiple edge nodes. In order to prevent information leakage, statistical calibration noise is usually added to the gradients in the model training process in combination with differential privacy (DP) technology. However, this privacy - protection measure inevitably reduces the accuracy of the model, because the local noise of edge learners and the global noise aggregation on the central server will all affect the performance of the model. ### Existing Challenges 1. **System Analysis**: It is necessary to systematically analyze the factors that affect local and aggregated noise. Besides the privacy budget, factors such as data sensitivity also need to be considered. 2. **Gradient Clipping Technology**: Existing differential privacy frameworks mainly rely on gradient clipping technology to calibrate local noise, but this method ignores the heterogeneity and dynamic changes of local gradients and the cumulative impact of these changes on accuracy. 3. **Overall Model Accuracy**: Existing technologies mainly focus on how to estimate the sensitivity of local noise, while ignoring how to reduce the negative impact of aggregated noise on the overall model accuracy. ### Solutions To address the above challenges, the paper proposes a new framework named PrivateDL, whose main features include: 1. **Sampling - Based Sensitivity Estimation**: By dynamically sampling the gradient range in each iteration round, the pre - defined gradient clipping boundaries are avoided, thereby reducing the local noise injected by each edge node. 2. **Virtual Batch Amplification**: By increasing the batch size to reduce the aggregated noise, and combined with redundant input data removal technology, the computational cost of gradient calculation is reduced. ### Technical Contributions - **System Analysis**: The paper analyzes the characteristics of distributed deep - learning model training and deduces the key factors that affect local and aggregated noise. - **Module Design**: Two PrivateDL modules are designed for efficient and accurate model training respectively: - **Sensitivity Estimation Module**: By dynamically sampling the gradient range, the local noise injected by each edge node is reduced. - **Virtual Batch Amplification Module**: By increasing the batch size to reduce the aggregated noise, and combined with redundant input data removal technology, the computational cost is reduced. - **Implementation and Evaluation**: PrivateDL is implemented on the KubeEdge platform and integrated with the deep - learning algorithms of Intel BigDL and PyTorch. Experimental results show that PrivateDL can significantly improve the accuracy of the model under different deep - learning training settings. ### Conclusion Through systematic analysis and design, PrivateDL can significantly improve the accuracy of deep - learning models on edge devices while ensuring data privacy. This provides a new solution for efficient federated learning in the edge - computing environment.

Accurate Differentially Private Deep Learning on the Edge

A Distributed Privacy-Preserving Framework for Deep Learning with Edge-Cloud Computing.

Privacy-Preserving Federated Edge Learning: Modeling and Optimization

Differentially Private Deep Learning With Dynamic Privacy Budget Allocation and Adaptive Optimization

Differentially-Private Deep Learning with Directional Noise

An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning

DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

Differentially Private Federated Learning with an Adaptive Noise Mechanism

GReDP: A More Robust Approach for Differential Private Training with Gradient-Preserving Noise Reduction

Learning from Differentially Private Neural Activations with Edge Computing

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

A Fine-Grained Differentially Private Federated Learning Against Leakage from Gradients

Asynchronous Federated Learning with Differential Privacy for Edge Intelligence

Mitigating Disparate Impact on Model Accuracy in Differentially Private Learning.

Locally Differentially Private Distributed Deep Learning via Knowledge Distillation

Differentially-Private Deep Learning from an Optimization Perspective.

A Two-Stage Differential Privacy Scheme for Federated Learning Based on Edge Intelligence

How to Democratise and Protect AI: Fair and Differentially Private Decentralised Deep Learning

A privacy-preserving deep learning approach for face recognition with edge computing

Differentially Private Federated Learning with Laplacian Smoothing