Abstract:Deep neural networks (DNNs) with noisy weights, which we refer to as noisy neural networks (NoisyNNs), arise from the training and inference of DNNs in the presence of noise. NoisyNNs emerge in many new applications, including the wireless transmission of DNNs, the efficient deployment or storage of DNNs in analog devices, and the truncation or quantization of DNN weights. This article studies a fundamental problem of NoisyNNs: how to reconstruct the DNN weights from their noisy manifestations. While prior works relied exclusively on the maximum likelihood (ML) estimation, this article puts forth a denoising approach to reconstruct DNNs with the aim of maximizing the inference accuracy of the reconstructed models. The superiority of our denoiser is rigorously proven in two small-scale problems, wherein we consider a quadratic neural network function and a shallow feedforward neural network, respectively. When applied to advanced learning tasks with modern DNN architectures, our denoiser exhibits significantly better performance than the ML estimator. Consider the average test accuracy of the denoised DNN model versus the weight variance to noise power ratio (WNR) performance. When denoising a noisy ResNet34 model arising from noisy inference, our denoiser outperforms ML estimation by up to 4.1 dB to achieve a test accuracy of 60%. When denoising a noisy ResNet18 model arising from noisy training, our denoiser outperforms ML estimation by 13.4 dB and 8.3 dB to achieve test accuracies of 60% and 80%, respectively.

High-dimensional asymptotics of denoising autoencoders *

High-dimensional Asymptotics of Denoising Autoencoders

Denoising Adversarial Autoencoders

Rate-Optimal Denoising with Deep Neural Networks

Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustness

Analyzing noise in autoencoders and deep networks

Stacked autoencoders based machine learning for noise reduction and signal reconstruction in geophysical data

Elastic Net with Adaptive Weight for Image Denoising

Denoising without access to clean data using a partitioned autoencoder

Gabor-based learnable sparse representation for self-supervised denoising

Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods

Why should autoencoders work?

Lateral Connections in Denoising Autoencoders Support Supervised Learning

Image Denoising Using Very Deep Fully Convolutional Encoder-Decoder Networks with Symmetric Skip Connections.

Thinner Latent Spaces: Detecting dimension and imposing invariance through autoencoder gradient constraints

Research of Stacked Denoising Sparse Autoencoder.

Learning Model-Blind Temporal Denoisers without Ground Truths

Denoising Noisy Neural Networks: A Bayesian Approach with Compensation.