Abstract:Current random feature models typically rely on fixed activation functions, limiting their ability to capture diverse patterns in data. To address this, we introduce the Random Feature model with Learnable Activation Functions (RFLAF), a novel model that significantly enhances the expressivity and interpretability of traditional random feature (RF) models. We begin by studying the RF model with a single radial basis function, where we discover a new kernel and provide the first theoretical analysis on it. By integrating the basis functions with learnable weights, we show that RFLAF can represent a broad class of random feature models whose activation functions belong in $C_c(\mathbb{R})$. Theoretically, we prove that the model requires only about twice the parameter number compared to a traditional RF model to achieve the significant leap in expressivity. Experimentally, RFLAF demonstrates two key advantages: (1) it performs better across various tasks compared to traditional RF model with the same number of parameters, and (2) the optimized weights offer interpretability, as the learned activation function can be directly inferred from these weights. Our model paves the way for developing more expressive and interpretable frameworks within random feature models.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations brought by the fixed activation function in traditional Random Feature Models. Specifically, since the activation function is not trainable, these models lack flexibility when fitting data and are unable to automatically find the optimal activation function to meet the requirements of different tasks. This restricts the model's ability to capture diverse patterns in the data. To solve this problem, the author proposes the Random Feature model with Learnable Activation Functions (RFLAF). This new model significantly enhances the model's expressiveness and interpretability by introducing learnable weights to integrate the basis functions. The main contributions of the paper are as follows: 1. **Theoretical analysis**: The author conducts an in - depth theoretical analysis of RFLAF, especially studying the special case of a single Radial Basis Function (RBF) activation, and discovers a new kernel function. They provide the analytical form of this kernel function and the first theoretical analysis of its representation and approximation characteristics. 2. **Parameter efficiency**: It is theoretically proven that RFLAF can achieve a significant improvement in expressiveness with only approximately twice the number of parameters as traditional RF models. In addition, the number of random features only needs to grow in proportion to the square root of the sample size. 3. **Experimental verification**: The experimental results show that RFLAF not only outperforms traditional RF models with the same number of parameters on various tasks, but also can directly infer the learned activation function through the optimized weights, providing higher interpretability. The model also exhibits the characteristics of being easy to implement and fast convergence. In summary, this paper aims to enhance the expressiveness and interpretability of random feature models by introducing learnable activation functions, so as to better adapt to different learning tasks.

Random Feature Models with Learnable Activation Functions

An Insect-Inspired Randomly, Weighted Neural Network with Random Fourier Features For Neuro-Symbolic Relational Learning

Random vector functional link network with subspace-based local connections

Random vector functional link network for short-term electricity load demand forecasting

Random Features Outperform Linear Models: Effect of Strong Input-Label Correlation in Spiked Covariance Data

Linear Complexity Randomized Self-attention Mechanism

Scalable Random Feature Latent Variable Models

Properties and Potential Applications of Random Functional-Linked Types of Neural Networks

Latent variable modeling with random features

A Duality Framework for Generalization Analysis of Random Feature Models and Two-Layer Neural Networks.

Error Bounds for Learning with Vector-Valued Random Features

Federated Spectrum Learning for Reconfigurable Intelligent Surfaces-Aided Wireless Edge Networks

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Optimal Kernel Quantile Learning with Random Features

Trans-dimensional Random Fields for Language Modeling.

Random vector functional link network: Recent developments, applications, and future directions

Wave-RVFL: A Randomized Neural Network Based on Wave Loss Function

A duality framework for analyzing random feature and two-layer neural networks

Universal approximation property of Banach space-valued random feature models including random neural networks

How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer

Variance-Reducing Couplings for Random Features