Output Range Analysis for Deep Neural Networks based on Simulated Annealing Processes

Helder Rojas,Nilton Rojas,Espinoza J. B.,Luis Huamanchumo

2024-07-03

Abstract:This paper tackles the challenging problem of output range estimation for Deep Neural Networks (DNNs), introducing a novel algorithm based on Simulated Annealing (SA). Our approach addresses the lack of local geometric information and high non-linearity in DNNs, making it versatile across various architectures, especially Residual Neural Networks (ResNets). We present a straightforward, implementation-friendly algorithm that avoids restrictive assumptions about network architecture. Through theoretical analysis and experimental evaluations, including tests on the Ackley function, we demonstrate our algorithm's effectiveness in navigating complex, non-convex surfaces and accurately estimating DNN output ranges. Futhermore, the Python codes of this experimental evaluation that support our results are available in our GitHub repository (<a class="link-external link-https" href="https://github.com/Nicerova7/output-range-analysis-for-deep-neural-networks-with-simulated-annealing" rel="external noopener nofollow">this https URL</a>).

Machine Learning,Probability

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the output range estimation problem of deep neural networks (DNNs). Specifically, the author focuses on how to effectively estimate the output range of DNNs in the absence of local geometric information and under high non - linearity conditions. This problem is crucial for ensuring the safety of DNNs in critical applications, such as self - driving cars and air traffic systems. Since DNNs are usually regarded as "black boxes", that is, their internal operating mechanisms are difficult to understand, this problem becomes particularly challenging. Existing methods often make strict assumptions about network architectures, which limits their application scope. For this reason, this paper proposes a new algorithm based on Simulated Annealing (SA). This algorithm can be applied to various neural network architectures, especially Residual Neural Networks (ResNets), and does not require strict assumptions about network architectures. Through theoretical analysis and experimental evaluation, the author shows the effectiveness of this algorithm in dealing with complex, non - convex surfaces and can accurately estimate the output range of DNNs.

Output Range Analysis for Deep Neural Networks based on Simulated Annealing Processes

Divide and Slide: Layer-Wise Refinement for Output Range Analysis of Deep Neural Networks

Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks

Self-adaptive deep neural network: Numerical approximation to functions and PDEs

Neighborhood Region Smoothing Regularization for Finding Flat Minima In Deep Neural Networks

Using Deep Neural Networks to Automate Large Scale Statistical Analysis for Big Data Applications

Finding Input Characterizations for Output Properties in ReLU Neural Networks

Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network

Neighborhood Region Smoothing Regularization for Finding Flat Minima in Deep Neural Networks

In-Depth Case Study on Artificial Neural Network Weights Optimization Using Meta-Heuristic and Heuristic Algorithmic Approach

DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems

Neural Operators for Accelerating Scientific Simulations and Design

NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks

A novel adaptive dynamic GA combined with AM to optimize ANN for multi-output prediction: small samples enhanced in industrial processing

A Self-adaptive neuroevolution approach to constructing Deep Neural Network architectures across different types

Performance Analysis of Deep Neural Networks for Direction of Arrival Estimation of Multiple Sources

NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization

Automated Construction of Neural Network Potential Energy Surface: The Enhanced Self-Organizing Incremental Neural Network Deep Potential Method

Novel Chiller Fault Diagnosis Using Deep Neural Network (DNN) with Simulated Annealing (SA)

An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters

Iterative Assessment and Improvement of DNN Operational Accuracy