Abstract:Deep Learning and Machine Learning models have recently been shown to be effective in many real world applications. While these models achieve increasingly better predictive performance, their structures have also become much more complex. A common and difficult problem for complex models is overfitting. Regularization is used to penalize the complexity of the model in order to avoid overfitting. However, in most learning frameworks, regularization function is usually set as some hyper parameters, and therefore the best setting is difficult to find. In this paper, we propose an adaptive regularization method, as part of a large end-to-end healthcare data analytics software stack, which effectively addresses the above difficulty. First, we propose a general adaptive regularization method based on Gaussian Mixture (GM) to learn the best regularization function according to the observed parameters. Second, we develop an effective update algorithm which integrates Expectation Maximization (EM) with Stochastic Gradient Descent (SGD). Third, we design a lazy update algorithm to reduce the computational cost by 4x. The overall regularization framework is fast, adaptive and easy-to-use. We validate the effectiveness of our regularization method through an extensive experimental study over 13 standard benchmark datasets and three kinds of deep learning/machine learning models. The results illustrate that our proposed adaptive regularization method achieves significant improvement over state-of-the-art regularization methods.

Regularized EM Algorithms: A Unified Framework and Statistical Guarantees

Regularized EM algorithm

Regularized EM algorithm for sparse parameter estimation in nonlinear dynamic systems with application to gene regulatory network inference

Statistical analysis for a penalized EM algorithm in high-dimensional mixture linear regression model

High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality

Precise Error Analysis of Regularized M-estimators in High-dimensions

The use of the EM algorithm for regularization problems in high-dimensional linear mixed-effects models

Separating biological variance from noise by applying EM algorithm to modified General Linear Model

Regularization in High-Dimensional Regression and Classification via Random Matrix Theory

A Unified Dynamic Approach to Sparse Model Selection

Convergence Theorems for Generalized Alternating Minimization Procedures.

Regularized TLS-EM for estimating missing data

Convergence of Expectation-Maximization Algorithm with Mixed-Integer Optimization

Adaptive Lightweight Regularization Tool for Complex Analytics

Exploiting regularity in sparse Generalized Linear Models

High-Dimensional Econometrics and Regularized GMM

Exploring an EM-algorithm for banded regression in computational neuroscience

Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Models

Stochastic Expectation Maximization with Variance Reduction.

A Statistical Theory of Regularization-Based Continual Learning