Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression

Aku Kammonen,Anamika Pandey,Erik von Schwerin,Raúl Tempone
2024-10-09
Abstract:This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in "Adaptive Random Fourier Features with Metropolis Sampling", Kammonen et al., Foundations of Data Science, 2(3):309--332, 2020. This improved method uses a particle filter type resampling technique to stabilize the training process and reduce sensitivity to parameter choices. With resampling, the Metropolis test may also be omitted, reducing the number of hyperparameters and reducing the computational cost per iteration, compared to ARFF. We present comprehensive numerical experiments demonstrating the efficacy of our proposed algorithm in function regression tasks, both as a standalone method and as a pre-training step before gradient-based optimization, here Adam. Furthermore, we apply our algorithm to a simple image regression problem, showcasing its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons (MLPs). In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.
Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to improve the Adaptive Random Fourier Features (ARFF) training algorithm by introducing the resampling technique, thereby enhancing the performance of shallow neural networks in function regression tasks. Specifically, the researchers hope: 1. **Stabilize the training process**: By using the particle - filtering - type resampling technique, reduce the sensitivity to parameter selection and make the training process more stable. 2. **Reduce the number of hyper - parameters**: By omitting the Metropolis test step, reduce the number of hyper - parameters and lower the computational cost per iteration. 3. **Increase the convergence speed of the initial iterations**: Especially when used as a pre - training step for gradient optimization methods such as Adam, it can reduce errors more quickly in the first few iterations. In addition, this paper also demonstrates the application of the proposed algorithm in the image regression problem, especially for the automated sampling of Random Fourier Feature (RFF) layer parameters in coordinate - based MLPs. ### Formula Summary - **Neural Network Model**: \[ \beta_{\theta}(x)=\sum_{k = 1}^{K}a_{k}\exp(i\omega_{k}^{T}x) \] where \(a_{k}\in\mathbb{C}\) and \(\omega_{k}\in\mathbb{R}^{d}\) represent the amplitude and frequency respectively. - **Regularized Least - Squares Optimization Problem**: \[ \min_{\theta\in\Theta}\left\{\mathbb{E}_{\rho}[(y - \beta_{\theta}(x))^{2}]+\lambda\|a\|_{2}^{2}\right\} \] where \(\lambda\geq0\) is the regularization parameter, and \(\|a\|_{2}^{2}=\sum_{k = 1}^{K}|a_{k}|^{2}\). - **Effective Sample Size (ESS)**: \[ K_{\text{ESS}}:=\left(\sum_{k = 1}^{K}\tilde{p}_{k}^{2}\right)^{-1} \] where \(\tilde{p}_{k}:=\frac{|a_{k}|}{\|a\|_{1}}\). ### Main Contributions 1. Proposed an improved Adaptive Random Fourier Features training algorithm based on resampling, which improves the stability of training and reduces the sensitivity to parameter selection. 2. Demonstrated the numerical experiment results of this algorithm in function regression tasks, including as an independent training algorithm and as a pre - training step for gradient optimization methods. 3. Applied this algorithm to the image regression problem and demonstrated its effectiveness in sampling RFF layer parameters. Through these improvements, the researchers aim to enhance the performance and robustness of the ARFF algorithm in practical applications.