Abstract:Building highly non-linear and non-parametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this article, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the DKMO (Deep Kernel Machine Optimization) framework, that creates an ensemble of dense embeddings using Nystrom kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pre-trained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques.

Kernel Sum of Squares for Data Adapted Kernel Learning of Dynamical Systems from Data: A global optimization approach

A Gradient-based Kernel Optimization Approach for Parabolic Distributed Parameter Control Systems

Learning dynamical systems from data: A simple cross-validation perspective, Part V: Sparse Kernel Flows for 132 chaotic dynamical systems

TKS-BLS: Temporal Kernel Stationary Broad Learning System for Enhanced Modeling, Anomaly Detection, and Incremental Learning with Application to Ironmaking Processes

Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series

Kernel optimization-based discriminant analysis for face recognition

Study of Set-Membership Adaptive Kernel Algorithms

Learning dynamical systems from data: Gradient-based dictionary optimization

Nonlinear Dynamic Process Monitoring Based on Kernel Partial Least Squares

Approximation of optimization problems with constraints through kernel Sum-Of-Squares

Generalized Semisupervised Self-Optimizing Kernel Model for Quality-Related Industrial Process Monitoring.

An Online Multiple Kernel Parallelizable Learning Scheme

Data-Driven Optimal Feedback Laws via Kernel Mean Embeddings

On Few Shot Learning of Dynamical Systems: A Koopman Operator Theoretic Approach

Data-Driven Stochastic Optimal Control Using Kernel Gradients

A Gradient-Based Optimization Method Using the Koopman Operator

Optimizing Kernel Machines using Deep Learning

Kernel Regression-Based State-Space Estimation of Continuous-Time Dynamic Systems From Noisy Sampled State Data

Multistage Stochastic Optimization via Kernels

Auxiliary Functions as Koopman Observables: Data-Driven Analysis of Dynamical Systems via Polynomial Optimization