Abstract:Kernel based methods yield approximation models that are flexible, efficient and powerful. In particular, they utilize fixed feature maps of the data, being often associated to strong analytical results that prove their accuracy. On the other hand, the recent success of machine learning methods has been driven by deep neural networks (NNs). They achieve a significant accuracy on very high-dimensional data, in that they are able to learn also efficient data representations or data-based feature maps. In this paper, we leverage a recent deep kernel representer theorem to connect the two approaches and understand their interplay. In particular, we show that the use of special types of kernels yield models reminiscent of neural networks that are founded in the same theoretical framework of classical kernel methods, while enjoying many computational properties of deep neural networks. Especially the introduced Structured Deep Kernel Networks (SDKNs) can be viewed as neural networks with optimizable activation functions obeying a representer theorem. Analytic properties show their universal approximation properties in different asymptotic regimes of unbounded number of centers, width and depth. Especially in the case of unbounded depth, the constructions is asymptotically better than corresponding constructions for ReLU neural networks, which is made possible by the flexibility of kernel approximation

Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks

Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory

Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines

Extraction of nonlinearity in neural networks with Koopman operator

A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks

Deep Koopman-layered Model with Universal Property Based on Toeplitz Matrices

Deep Karaoke: Extracting Vocals from Musical Mixtures Using a Convolutional Deep Neural Network

Universality and Optimality of Structured Deep Kernel Networks

Learning Compositional Koopman Operators for Model-Based Control

Reconstructing Voice Identity from Noninvasive Auditory Cortex Recordings

Distributed Deep Koopman Learning for Nonlinear Dynamics

Dissecting neural computations in the human auditory pathway using deep neural networks for speech

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study

Deep Complex Networks

Research on deep neural network's hidden layers in phoneme recognition

Speech Reconstruction Using a Deep Partially Supervised Neural Network

Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems

Dynamical systems and complex networks: A Koopman operator perspective

Autoencoding for the 'Good Dictionary' of eigen pairs of the Koopman Operator