Abstract:The need for scalable and expressive models in machine learning is paramount, particularly in applications requiring both structural depth and flexibility. Traditional deep learning methods, such as multilayer perceptrons (MLP), offer depth but lack ability to integrate structural characteristics of deep learning architectures with non-parametric flexibility of kernel methods. To address this, deep kernel learning (DKL) was introduced, where inputs to a base kernel are transformed using a deep learning architecture. These kernels can replace standard kernels, allowing both expressive power and scalability. The advent of Kolmogorov-Arnold Networks (KAN) has generated considerable attention and discussion among researchers in scientific domain. In this paper, we introduce a scalable deep kernel using KAN (DKL-KAN) as an effective alternative to DKL using MLP (DKL-MLP). Our approach involves simultaneously optimizing these kernel attributes using marginal likelihood within a Gaussian process framework. We analyze two variants of DKL-KAN for a fair comparison with DKL-MLP: one with same number of neurons and layers as DKL-MLP, and another with approximately same number of trainable parameters. To handle large datasets, we use kernel interpolation for scalable structured Gaussian processes (KISS-GP) for low-dimensional inputs and KISS-GP with product kernels for high-dimensional inputs. The efficacy of DKL-KAN is evaluated in terms of computational training time and test prediction accuracy across a wide range of applications. Additionally, the effectiveness of DKL-KAN is also examined in modeling discontinuities and accurately estimating prediction uncertainty. The results indicate that DKL-KAN outperforms DKL-MLP on datasets with a low number of observations. Conversely, DKL-MLP exhibits better scalability and higher test prediction accuracy on datasets with large number of observations.

KLANN: Linearising Long-Term Dynamics in Nonlinear Audio Effects Using Koopman Networks

Comparative Study of State-based Neural Networks for Virtual Analog Audio Effects Modeling

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory

Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic System Identification with Application to Audio Processing

Extraction of nonlinearity in neural networks with Koopman operator

Hyper Recurrent Neural Network: Condition Mechanisms for Black-box Audio Effect Modeling

CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixels

Koopman analysis of nonlinear systems with a neural network representation

Koopman dynamic-oriented deep learning for invariant subspace identification and full-state prediction of complex systems

Exploring Quality and Generalizability in Parameterized Neural Audio Effects

DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks

Modulation Extraction for LFO-driven Audio Effects

NeuralKalman: A Learnable Kalman Filter for Acoustic Echo Cancellation

Leveraging KANs For Enhanced Deep Koopman Operator Discovery

Machine learning-based input-augmented Koopman modeling and predictive control of nonlinear processes

Data-driven modelling of brain activity using neural networks, Diffusion Maps, and the Koopman operator

Identification of MIMO Wiener-type Koopman models for data-driven model reduction using deep learning

Invertible Koopman Network and Its Application in Data-Driven Modeling for Dynamic Systems

Data-driven Nonlinear Model Reduction using Koopman Theory: Integrated Control Form and NMPC Case Study

Differentiable Signal Processing With Black-Box Audio Effects

A convolutional neural-network model of human cochlear mechanics and filter tuning for real-time applications