Abstract:This paper is a preliminary study of the robustness and noise analysis of deep neural networks via a game theory formulation Bayesian Neural Networks (BNN) and the maximal coding rate distortion loss. BNN has been shown to provide some robustness to deep learning, and the minimax method used to be a natural conservative way to assist the Bayesian method. Inspired by the recent closed-loop transcription neural network, we formulate the BNN via game theory between the deterministic neural network $f$ and the sampling network $f + \xi$ or $f + r*\xi$. Compared with previous BNN, BNN via game theory learns a solution space within a certain gap between the center $f$ and the sampling point $f + r*\xi$, and is a conservative choice with a meaningful prior setting compared with previous BNN. Furthermore, the minimum points between $f$ and $f + r*\xi$ become stable when the subspace dimension is large enough with a well-trained model $f$. With these, the model $f$ can have a high chance of recognizing the out-of-distribution data or noise data in the subspace rather than the prediction level, even if $f$ is in online training after a few iterations of true data. So far, our experiments are limited to MNIST and Fashion MNIST data sets, more experiments with realistic data sets and complicated neural network models should be implemented to validate the above arguments.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the robustness of deep neural networks (DNN) in the face of noise and out - of - distribution data. Specifically, by introducing a Bayesian neural network (Bayesian Neural Networks, BNN) based on a game - theoretic framework, the author aims to improve the robustness of DNN and its tolerance to noise.
### Main problems:
1. **Robustness of deep neural networks**: Although DNN has achieved success in many fields, its vulnerability in the face of adversarial attacks, noise interference, and out - of - distribution data remains a problem that urgently needs to be solved.
2. **Uncertainty quantification**: Traditional DNNs are difficult to effectively quantify the uncertainty of the model, which limits their reliability in some key applications.
3. **Conservatism of prior settings**: How to select an appropriate prior distribution for BNN to ensure the stability and robustness of the model in different situations.
### Solutions:
To address the above problems, the paper proposes a BNN framework based on game theory (MinMax BNN) and optimizes the model through the Maximal Coding Rate Distortion (MCR) loss function. The main contributions of this framework include:
1. **New BNN framework**: A new BNN framework is constructed through game theory. This framework can learn a solution space, in which there is a certain gap between the central point \( f \) and the sampling point \( f + r * \xi \). This gap helps to improve the robustness of the model and its tolerance to noise.
2. **Conservative prior settings**: Compared with traditional BNNs, this framework provides a more conservative prior - setting reference, making the model more stable in the face of uncertainty and noise.
3. **Detecting out - of - distribution data and noise data**: Even when the model is not fully trained, this framework can detect out - of - distribution data and noise data in the subspace, thereby improving the security and reliability of the model.
### Formula representation:
The following formula is used in the paper to describe the optimization objective of Minimax BNN:
\[
\min_{\rho, r} \max_{\mu} \tau(\mu, \rho, r)=\Delta R(f(X, \mu))+\Delta R(h(X, \mu, \rho, r))+\sum_{i = 1}^{k}\Delta R(f(X, \mu), h(X, \mu, \rho, r))
\]
where:
- \( X \) represents the input data;
- \( f \) represents the deterministic network or the central point of MinMax BNN;
- \( h(X, \mu, \rho)=f + r * \xi \) represents the sampling network;
- \( \mu \) represents the weights of the deterministic network \( f(X, \mu) \);
- \( \rho \) represents the randomness or variance shape;
- \( r \) is a proportional parameter determined by the loss function;
- \( k \) represents the number of classes;
- \( \tau(\mu, \rho, r) \) represents the objective function using MCR;
- \( \Delta R(f(X, \mu)) \) represents the MCR calculated by \( f \);
- \( \Delta R(h(X, \mu, \rho)) \) represents the MCR calculated by the sampling network \( f + r * \xi \);
- \( \sum_{i = 1}^{k}\Delta R(f(X, \mu), h(X, \mu, \rho, r)) \) calculates the "distance" between \( f \) and \( f + r * \xi \).
In this way, the paper not only proposes a new theoretical framework but also verifies its effectiveness on the MNIST and Fashion MNIST datasets through experiments.