Abstract:Formal certification of Neural Networks (NNs) is crucial for ensuring their safety, fairness, and robustness. Unfortunately, on the one hand, sound and complete certification algorithms of ReLU-based NNs do not scale to large-scale NNs. On the other hand, incomplete certification algorithms are easier to compute, but they result in loose bounds that deteriorate with the depth of NN, which diminishes their effectiveness. In this paper, we ask the following question; can we replace the ReLU activation function with one that opens the door to incomplete certification algorithms that are easy to compute but can produce tight bounds on the NN's outputs? We introduce DeepBern-Nets, a class of NNs with activation functions based on Bernstein polynomials instead of the commonly used ReLU activation. Bernstein polynomials are smooth and differentiable functions with desirable properties such as the so-called range enclosure and subdivision properties. We design a novel algorithm, called Bern-IBP, to efficiently compute tight bounds on DeepBern-Nets outputs. Our approach leverages the properties of Bernstein polynomials to improve the tractability of neural network certification tasks while maintaining the accuracy of the trained networks. We conduct comprehensive experiments in adversarial robustness and reachability analysis settings to assess the effectiveness of the proposed Bernstein polynomial activation in enhancing the certification process. Our proposed framework achieves high certified accuracy for adversarially-trained NNs, which is often a challenging task for certifiers of ReLU-based NNs. Moreover, using Bern-IBP bounds for certified training results in NNs with state-of-the-art certified accuracy compared to ReLU networks. This work establishes Bernstein polynomial activation as a promising alternative for improving NN certification tasks across various applications.

Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement

Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation

On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks

Accurate Low-Degree Polynomial Approximation of Non-polynomial Operators for Fast Private Inference in Homomorphic Encryption

Batch-oriented Element-wise Approximate Activation for Privacy-Preserving Neural Networks

Provable Preimage Under-Approximation for Neural Networks (Full Version)

Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Why Deep Neural Networks for Function Approximation?

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks

Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits

Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Time

Ax-BxP: Approximate Blocked Computation for Precision-Reconfigurable Deep Neural Network Acceleration

A Priori Error Estimation of Physics-Informed Neural Networks Solving Allen--Cahn and Cahn--Hilliard Equations

Loss-attentional physics-informed neural networks

Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?

DeepBern-Nets: Taming the Complexity of Certifying Neural Networks using Bernstein Polynomial Activations and Precise Bound Propagation

Memristor-Based Neuromorphic Hardware Improvement for Privacy-Preserving ANN

Improving PINNs By Algebraic Inclusion of Boundary and Initial Conditions