Abstract:All fields of science depend on mathematical models. Occam's razor refers to the principle that good models should exclude parameters beyond those minimally required to describe the systems they represent. This is because redundancy can lead to incorrect estimates of model parameters from data, and thus inaccurate or ambiguous conclusions. Here, we show how deep learning can be powerfully leveraged to apply Occam's razor to model parameters. Our method, FixFit, uses a feedforward deep neural network with a bottleneck layer to characterize and predict the behavior of a given model from its input parameters. FixFit has three major benefits. First, it provides a metric to quantify the original model's degree of complexity. Second, it allows for the unique fitting of data. Third, it provides an unbiased way to discriminate between experimental hypotheses that add value versus those that do not. In three use cases, we demonstrate the broad applicability of this method across scientific domains. To validate the method using a known system, we apply FixFit to recover known composite parameters for the Kepler orbit model and a dynamic model of blood glucose regulation. In the latter, we demonstrate the ability to fit the latent parameters to real data. To illustrate how the method can be applied to less well-established fields, we use it to identify parameters for a multi-scale brain model and reduce the search space for viable candidate mechanisms. Mathematical modeling is a pillar of scientific inquiry, bridging the gap between theory and experimental observations. However, in complex systems such as those pervasive in biology (e.g., gene regulatory networks, multi-scale brain interactions, and drug pharmacokinetics), different mechanisms can yield equally plausible explanations for the data. This ambiguity is not due to data limitations but rather to the equations that govern these systems. Both the interpretation and estimation of the parameters of these models are hindered by these intrinsic degeneracies. For this reason, we present a general tool that harnesses the power of deep learning to automatically identify and rectify these ambiguities, allowing a broader range of models to be precisely determined by experimental data. We show, for example, that our method provides novel insights into the features of multi-scale brain dynamics that can be learned from functional neuroimaging. This represents a crucial initial step toward characterizing the mechanistic insights that different types of experiments can provide.

A resource-efficient model for deep kernel learning

Optimizing Kernel Machines using Deep Learning

How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

Deep Clustered Convolutional Kernels

Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies

Universality and Optimality of Structured Deep Kernel Networks

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

Shared Deep Kernel Learning for Dimensionality Reduction.

On Kernel Method-Based Connectionist Models and Supervised Deep Learning Without Backpropagation

Learning Explicit Deep Representations from Deep Kernel Networks

Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training

Efficient kernel surrogates for neural network-based regression

Scalable and Sustainable Deep Learning via Randomized Hashing

A Multilayered-and-Randomized Latent Factor Model for High-Dimensional and Sparse Matrices

DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks

Toward Large Kernel Models

Achieving Occam's razor: Deep learning for optimal model reduction

Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics

Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions

Sparse kernel deep stacking networks

A parametric framework for kernel-based dynamic mode decomposition using deep learning