Abstract:This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in "Adaptive Random Fourier Features with Metropolis Sampling", Kammonen et al., Foundations of Data Science, 2(3):309--332, 2020. This improved method uses a particle filter type resampling technique to stabilize the training process and reduce sensitivity to parameter choices. With resampling, the Metropolis test may also be omitted, reducing the number of hyperparameters and reducing the computational cost per iteration, compared to ARFF. We present comprehensive numerical experiments demonstrating the efficacy of our proposed algorithm in function regression tasks, both as a standalone method and as a pre-training step before gradient-based optimization, here Adam. Furthermore, we apply our algorithm to a simple image regression problem, showcasing its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons (MLPs). In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to improve the Adaptive Random Fourier Features (ARFF) training algorithm by introducing the resampling technique, thereby enhancing the performance of shallow neural networks in function regression tasks. Specifically, the researchers hope: 1. **Stabilize the training process**: By using the particle - filtering - type resampling technique, reduce the sensitivity to parameter selection and make the training process more stable. 2. **Reduce the number of hyper - parameters**: By omitting the Metropolis test step, reduce the number of hyper - parameters and lower the computational cost per iteration. 3. **Increase the convergence speed of the initial iterations**: Especially when used as a pre - training step for gradient optimization methods such as Adam, it can reduce errors more quickly in the first few iterations. In addition, this paper also demonstrates the application of the proposed algorithm in the image regression problem, especially for the automated sampling of Random Fourier Feature (RFF) layer parameters in coordinate - based MLPs. ### Formula Summary - **Neural Network Model**: \[ \beta_{\theta}(x)=\sum_{k = 1}^{K}a_{k}\exp(i\omega_{k}^{T}x) \] where \(a_{k}\in\mathbb{C}\) and \(\omega_{k}\in\mathbb{R}^{d}\) represent the amplitude and frequency respectively. - **Regularized Least - Squares Optimization Problem**: \[ \min_{\theta\in\Theta}\left\{\mathbb{E}_{\rho}[(y - \beta_{\theta}(x))^{2}]+\lambda\|a\|_{2}^{2}\right\} \] where \(\lambda\geq0\) is the regularization parameter, and \(\|a\|_{2}^{2}=\sum_{k = 1}^{K}|a_{k}|^{2}\). - **Effective Sample Size (ESS)**: \[ K_{\text{ESS}}:=\left(\sum_{k = 1}^{K}\tilde{p}_{k}^{2}\right)^{-1} \] where \(\tilde{p}_{k}:=\frac{|a_{k}|}{\|a\|_{1}}\). ### Main Contributions 1. Proposed an improved Adaptive Random Fourier Features training algorithm based on resampling, which improves the stability of training and reduces the sensitivity to parameter selection. 2. Demonstrated the numerical experiment results of this algorithm in function regression tasks, including as an independent training algorithm and as a pre - training step for gradient optimization methods. 3. Applied this algorithm to the image regression problem and demonstrated its effectiveness in sampling RFF layer parameters. Through these improvements, the researchers aim to enhance the performance and robustness of the ARFF algorithm in practical applications.

Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression

Stein Random Feature Regression

Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedback

Adaptive Random Fourier Features Kernel LMS

RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features

End-to-end Kernel Learning via Generative Random Fourier Features

Random Fourier Features Extended Kernel Recursive Least P-Power Algorithm

Anisotropic Fourier Features for Neural Image-Based Rendering and Relighting

Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks

Accelerating Neural Field Training via Soft Mining

Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Features

Learning Random Fourier Features by Hybrid Constrained Optimization

Random fourier features for asymmetric kernels

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

Deep Learning without Global Optimization by Random Fourier Neural Networks

Deep Fourier-based Arbitrary-scale Super-resolution for Real-time Rendering

RL-based Stateful Neural Adaptive Sampling and Denoising for Real-Time Path Tracing

FouRA: Fourier Low Rank Adaptation

Slim-RFFNet: Slim deep convolution random Fourier feature network for image classification

Nyström Method vs Random Fourier Features: A Theoretical and Empirical Comparison.

Neural Fourier Filter Bank