Abstract:With the rapid development of deep learning in various fields of science and technology, such as speech recognition, image classification, and natural language processing, recently it is also widely applied in the functional data analysis (FDA) with some empirical success. However, due to the infinite dimensional input, we need a powerful dimension reduction method for functional learning tasks, especially for the nonlinear functional regression. In this paper, based on the idea of smooth kernel integral transformation, we propose a functional deep neural network with an efficient and fully data-dependent dimension reduction method. The architecture of our functional net consists of a kernel embedding step: an integral transformation with a data-dependent smooth kernel; a projection step: a dimension reduction by projection with eigenfunction basis based on the embedding kernel; and finally an expressive deep ReLU neural network for the prediction. The utilization of smooth kernel embedding enables our functional net to be discretization invariant, efficient, and robust to noisy observations, capable of utilizing information in both input functions and responses data, and have a low requirement on the number of discrete points for an unimpaired generalization performance. We conduct theoretical analysis including approximation error and generalization error analysis, and numerical simulations to verify these advantages of our functional net.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to effectively use deep neural networks for non - linear function regression when dealing with functional data. Specifically, since functional data is essentially infinite - dimensional, a powerful dimension - reduction method is required to process these data, especially in non - linear function regression tasks. The paper proposes a Functional Deep Neural Network (FDNN) based on Smooth Kernel Embedding, which contains three main steps: 1. **Kernel Embedding**: Transform the input function using a data - dependent smoothing kernel through integral transformation. This step makes the model invariant to discretization and can effectively handle noisy observations. 2. **Projection**: Achieve dimension reduction by projecting based on the eigenfunction basis of the embedding kernel. This step utilizes the eigenvalues and eigenfunctions of the kernel function, thus retaining the key information of the input function while reducing the dimension. 3. **Deep ReLU Neural Network**: Finally, use an expressive Deep ReLU Neural Network for prediction. Through this design, the proposed FDNN model can not only handle infinite - dimensional functional data, but also reduce the requirement for the number of discrete points while maintaining the generalization performance of the model. In addition, the model also performs well in theoretical analysis, including the analysis of approximation error and generalization error, and numerical simulations verify the advantages of the model. ### Main Contributions 1. **Discretization Invariance**: The model is invariant to data at different observation points, which means that even if the observation points change, there is no need to retrain the model. 2. **Noise Resistance**: As a pre - smoothing method, kernel embedding is robust to noisy observations of the input function and can generate smooth functions. 3. **Efficient Dimension Reduction**: By selecting the optimal eigenfunction basis for dimension reduction, the model can fully utilize the information of the input function and response data while maintaining computational efficiency. 4. **Theoretical Support**: The paper conducts a detailed theoretical analysis, including the analysis of approximation error and generalization error, proving the effectiveness and superiority of the model. ### Application Scenarios This model is particularly suitable for handling non - linear regression problems of functional data, such as curve or surface data analysis in biomedical, physics, and engineering fields. Through this method, useful information can be more accurately extracted from complex data, providing strong support for scientific research.

Nonlinear functional regression by functional deep neural network with kernel embedding

Functional data analysis using deep neural networks

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Estimation of the Mean Function of Functional Data via Deep Neural Networks

Deep Learning With Functional Inputs

Nonlinear Functional Principal Component Analysis Using Neural Networks

Interpreting and generalizing deep learning in physics-based problems with functional linear models

Kernel partial least squares regression for functional nonlinear models

Deep Neural Network Classifier for Multi-dimensional Functional Data

Partially Functional Linear Regression in High Dimensions

Functional Network Method for the Identification of Nonlinear Systems

Deep Learning for Functional Data Analysis with Adaptive Basis Layers

Functional autoencoder for smoothing and representation learning

Functional Nonlinear Learning

Multiclass classification for multidimensional functional data through deep neural networks

Functional Linear Regression with Mixed Predictors

Representation of Functional Data in Neural Networks

Approximation of RKHS Functionals by Neural Networks

Robust Deep Neural Network Estimation for Multi-dimensional Functional Data