Abstract:Sparse regression and feature extraction are the cornerstones of knowledge discovery from massive data. Their goal is to discover interpretable and predictive models that provide simple relationships among scientific variables. While the statistical tools for model discovery are well established in the context of linear regression, their generalization to nonlinear regression in material modeling is highly problem-specific and insufficiently understood. Here we explore the potential of neural networks for automatic model discovery and induce sparsity by a hybrid approach that combines two strategies: regularization and physical constraints. We integrate the concept of Lp regularization for subset selection with constitutive neural networks that leverage our domain knowledge in kinematics and thermodynamics. We train our networks with both, synthetic and real data, and perform several thousand discovery runs to infer common guidelines and trends: L2 regularization or ridge regression is unsuitable for model discovery; L1 regularization or lasso promotes sparsity, but induces strong bias; only L0 regularization allows us to transparently fine-tune the trade-off between interpretability and predictability, simplicity and accuracy, and bias and variance. With these insights, we demonstrate that Lp regularized constitutive neural networks can simultaneously discover both, interpretable models and physically meaningful parameters. We anticipate that our findings will generalize to alternative discovery techniques such as sparse and symbolic regression, and to other domains such as biology, chemistry, or medicine. Our ability to automatically discover material models from data could have tremendous applications in generative material design and open new opportunities to manipulate matter, alter properties of existing materials, and discover new materials with user-defined properties.

Flexible, non-parametric modeling using regularized neural networks

Data-adaptive additive modeling

Adaptive Noisy Data Augmentation for Regularized Estimation and Inference in Generalized Linear Models

PANDA: AdaPtive Noisy Data Augmentation for Regularization of Undirected Graphical Models

Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean

Netreg: Network-Regularized Linear Models for Biological Association Studies

Additive Nonparametric Regression in the Presence of Endogenous Regressors

Additive model building for spatial regression

Stochastic Gradient Descent for Nonparametric Regression

Improving Data Analytics with Fast and Adaptive Regularization

Proximal Iteration for Nonlinear Adaptive Lasso

Improving Neural Additive Models with Bayesian Principles

Additive regularization schedule for neural architecture search

On sparse regression, Lp-regularization, and automated model discovery

An amortized approach to non-linear mixed-effects modeling based on neural posterior estimation

Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification

L-1 Regularized Projection Pursuit For Additive Model Learning

Nonparametric Covariance Regression for Massive Neural Data on Restricted Covariates via Graph

Bayesian Modeling of Nonlinear Poisson Regression with Artificial Neural Networks

Exploring the balance between interpretability and performance with carefully designed constrainable neural additive models