KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling

Yufan Zhou,Changyou Chen,Jinhui Xu

DOI: https://doi.org/10.48550/arXiv.1912.00979

2020-06-25

Abstract:Learning with kernels is an important concept in machine learning. Standard approaches for kernel methods often use predefined kernels that require careful selection of hyperparameters. To mitigate this burden, we propose in this paper a framework to construct and learn a data-dependent kernel based on random features and implicit spectral distributions that are parameterized by deep neural networks. The constructed network (called KernelNet) can be applied to deep generative modeling in various scenarios, including two popular learning paradigms in deep generative models, MMD-GAN and implicit Variational Autoencoder (VAE). We show that our proposed kernel indeed exists in applications and is guaranteed to be positive definite. Furthermore, the induced Maximum Mean Discrepancy (MMD) can endow the continuity property in weak topology by simple regularization. Extensive experiments indicate that our proposed KernelNet consistently achieves better performance compared to related methods.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to reduce the dependence on predefined kernel functions and their hyper - parameter selection in deep generative modeling, thereby improving the performance of the model. Specifically, the author proposes a framework to construct and learn data - dependent kernel functions through random features and implicit spectral distributions parameterized by deep neural networks. This method aims to overcome the sub - optimal solution problems caused by manual selection of kernel functions and hyper - parameters in traditional kernel methods, and can be applied in multiple deep generative model scenarios, including MMD - GAN and implicit variational auto - encoder (VAE). The paper also proves that the proposed kernel function does exist and guarantees its positive definiteness. Meanwhile, through simple regularization, the induced maximum mean discrepancy (MMD) can be endowed with continuous properties in the weak topology. Experimental results show that, compared with related methods, the proposed KernelNet can achieve better performance.

KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling

A Unified Kernel for Neural Network Learning

Optimizing Kernel Machines using Deep Learning

Deep Latent-Variable Kernel Learning

Learning Data-adaptive Nonparametric Kernels

A General Kernelization Framework for Learning Algorithms Based on Kernel PCA.

Universality and Optimality of Structured Deep Kernel Networks

Learning a Data-Dependent Kernel Function for KPCA-based Nonlinear Process Monitoring

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

Guided Deep Kernel Learning

Model-based Transductive Learning of the Kernel Matrix

How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

The Deep Kernelized Autoencoder

Deep Context-Aware Kernel Networks

Generative Model with Kernel Density Estimation

Online Kernel based Generative Adversarial Networks

Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels

Generative Adversarial Learning via Kernel Density Discrimination

Instability and Local Minima in GAN Training with Kernel Discriminators

Kernel Mean Matching for Content Addressability of GANs

A Unified Framework for Kernelization: the Empirical Kernel Feature Space