Abstract:This paper introduces a novel theoretical framework for the analysis of vector-valued neural networks through the development of vector-valued variation spaces, a new class of reproducing kernel Banach spaces. These spaces emerge from studying the regularization effect of weight decay in training networks with activations like the rectified linear unit (ReLU). This framework offers a deeper understanding of multi-output networks and their function-space characteristics. A key contribution of this work is the development of a representer theorem for the vector-valued variation spaces. This representer theorem establishes that shallow vector-valued neural networks are the solutions to data-fitting problems over these infinite-dimensional spaces, where the network widths are bounded by the square of the number of training data. This observation reveals that the norm associated with these vector-valued variation spaces encourages the learning of features that are useful for multiple tasks, shedding new light on multi-task learning with neural networks. Finally, this paper develops a connection between weight-decay regularization and the multi-task lasso problem. This connection leads to novel bounds for layer widths in deep networks that depend on the intrinsic dimensions of the training data representations. This insight not only deepens the understanding of the deep network architectural requirements, but also yields a simple convex optimization method for deep neural network compression. The performance of this compression procedure is evaluated on various architectures.

Vector-valued Reproducing Kernel Banach Spaces with Applications to Multi-task Learning

Learning in RKHM: a $C^*$-Algebraic Twist for Kernel Machines

Eigenfunction-Based Multitask Learning in a Reproducing Kernel Hilbert Space.

Multi-task Vector Field Learning.

Refinement of Operator-valued Reproducing Kernels

Multi-Task Classification Hypothesis Space with Improved Generalization Bounds

Kernel Bayesian Inference with Posterior Regularization

Random Fourier Features for Operator-Valued Kernels

A reproducing kernel Hilbert space framework for spike train signal processing

Relative reproducing kernels in vector-valued Hilbert and Banach spaces

Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression

Duality in RKHSs with Infinite Dimensional Outputs: Application to Robust Losses

Multiple Operator-valued Kernel Learning

Multi-Task Kernel Null-Space for One-Class Classification

Mirror Descent on Reproducing Kernel Banach Spaces

Vector valued de Branges spaces of entire functions based on pairs of Fredholm operator valued functions and functional model

A survey on kernel-based multi-task learning

Neural reproducing kernel Banach spaces and representer theorems for deep networks

Sparse multiresolution representations with adaptive kernels

Regularization Strategies and Empirical Bayesian Learning for MKL

Robust kernel ensemble regression in diversified kernel space with shared parameters