Abstract:Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using ``highly precise'' approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and existing literature often determined the required approximation precision empirically. In this paper, we initiate the investigation of PANN as a standalone object. Specifically, our contribution is two-fold. Firstly, we provide an explanation on the effect of approximate error in PANN. In particular, we discovered that (1) PANN is susceptible to some type of perturbations; and (2) weight regularisation significantly reduces PANN's accuracy. We support our explanation with experiments. Secondly, based on the insights from our investigations, we propose solutions to increase inference accuracy for PANN. Experiments showed that combination of our solutions is very effective: at the same precision, our PANN is 10% to 50% more accurate than state-of-the-arts; and at the same accuracy, our PANN only requires a precision of 2^{-9} while state-of-the-art solution requires a precision of 2^{-12} using the ResNet-20 model on CIFAR-10 dataset.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the trade - off problem between accuracy and computational efficiency when using low - precision Polynomial Approximation of Neural Network (PANN) in Privacy - Preserving Machine Learning (PPML). Specifically, the author focuses on: 1. **The influence of approximation error on the inference accuracy of PANN**: - In the privacy - protection scenario, in order to ensure data privacy, non - polynomial functions (such as ReLU activation functions) are usually replaced by polynomial approximations. However, the error introduced by this approximation will have an impact on the inference accuracy of the model. - Most of the existing literature determines the required approximation accuracy empirically, and less is known about the specific impact of approximation error. 2. **How to improve the accuracy of low - precision PANN**: - The author proposes two solutions to enhance the "sturdiness" of the neural network, that is, to reduce the impact of approximation error on inference accuracy. These methods include: - **Negative - input leakage method**: By introducing perturbations to the negative inputs of ReLU, the loss caused by approximation error is reduced. - **Reducing weight regularization**: By minimizing weight regularization and combining with the Mixup technique, the sturdiness of PANN is significantly improved while maintaining the accuracy of the backbone model. 3. **Application of general model structures**: - Existing improvement methods usually need to be fine - tuned for specific PANNs, which increases the training cost and is not universal. The method proposed in this paper is applicable to various general model structures and different approximation accuracy requirements without retraining the model. ### Main contributions - **Theoretical analysis**: The author proves the influence of approximation error on PANN through theoretical analysis, especially the different performances of the ReLU function on negative and positive inputs. - **Experimental verification**: The correctness of the theoretical analysis is verified through extensive experiments, and the effectiveness of the proposed solutions under different model structures, data sets and approximation methods is demonstrated. - **Performance improvement**: The experimental results show that after combining the proposed solutions, the accuracy of PANN at the same precision is increased by 10% - 50%, and the required precision at the same accuracy is reduced from \(2^{-12}\) to \(2^{-9}\). ### Summary This paper proposes a new method to improve the accuracy of low - precision PANN by in - depth research on the approximation error problem in PANN, thus achieving higher efficiency and better performance in privacy - preserving machine learning.

Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement

Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation

On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks

Accurate Low-Degree Polynomial Approximation of Non-polynomial Operators for Fast Private Inference in Homomorphic Encryption

Batch-oriented Element-wise Approximate Activation for Privacy-Preserving Neural Networks

Provable Preimage Under-Approximation for Neural Networks (Full Version)

Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Why Deep Neural Networks for Function Approximation?

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks

Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits

Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Time

Ax-BxP: Approximate Blocked Computation for Precision-Reconfigurable Deep Neural Network Acceleration

A Priori Error Estimation of Physics-Informed Neural Networks Solving Allen--Cahn and Cahn--Hilliard Equations

Loss-attentional physics-informed neural networks

Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?

DeepBern-Nets: Taming the Complexity of Certifying Neural Networks using Bernstein Polynomial Activations and Precise Bound Propagation

Memristor-Based Neuromorphic Hardware Improvement for Privacy-Preserving ANN

Improving PINNs By Algebraic Inclusion of Boundary and Initial Conditions