Abstract:The Gauss Markov theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. The classical approach to designing non-linear MVUEs is through maximum likelihood estimation (MLE) which often involves computationally challenging optimizations. On the other hand, deep learning methods allow for non-linear estimators with fixed computational complexity. Learning based estimators perform optimally on average with respect to their training set but may suffer from significant bias in other parameters. To avoid this, we propose to add a simple bias constraint to the loss function, resulting in an estimator we refer to as Bias Constrained Estimator (BCE). We prove that this yields asymptotic MVUEs that behave similarly to the classical MLEs and asymptotically attain the Cramer Rao bound. We demonstrate the advantages of our approach in the context of signal to noise ratio estimation as well as covariance estimation. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance. Examples include distributed sensor networks and data augmentation in test-time. In such applications, we show that BCE leads to asymptotically consistent estimators.

On a Biased Prediction Based on Optimal Mean Square Error Criterion

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

Design of Two MUSIC-like Estimators Based on Bias Minimization

Performance Bound Approximation for Bearing Estimation with Bias Correction

M-estimator Based Support Vector Machine and Its Application

A new analysis of optimal estimation and prediction under linear mixed models

Alternative Mean Square Error Estimators and Confidence Intervals for Prediction of Nonlinear Small Area Parameters

The Superiority of Bayes Linear Unbiased Minimum Variance Estimator with Respect to Ridge Estimator

Non-parametric Shrinkage Mean Estimation for Quadratic Loss Functions with Unknown Covariance Matrices

Small-sample Properties for Generalized Ridge Estimator of Regression Coefficients in Linear Regression Model

Two-Stage Maximum Score Estimator

A Non-Parametric Shrinkage Mean Estimation for Arbitrary Quadratic Loss Functions and Unknown Covariance Matrices

A New Interpretation on the MMSE As a Robust MEE Criterion

Regularized Kernel Forms of Minimum Squared Error Method

A Bilateral Bound on the Mean-Square Error for Estimation in Model Mismatch

Learning to Estimate Without Bias

Mean Square Prediction Error of Misspecified Gaussian Process Models

Risk-Aware MMSE Estimation

Optimal designs for linear models with Fredholm-type errors

On Optimal Estimations with Minimum Error Entropy Criterion

Fixed-Point Minimum Error Entropy With Fiducial Points