Abstract:Deep neural networks have been extensively used for the solution of both the forward and the inverse problem for dynamical systems. However, their implementation necessitates optimizing a high-dimensional space of parameters and hyperparameters. This fact, along with the requirement of substantial computational resources, pose a barrier to achieving high numerical accuracy, but also interpretability. Here, to address the above challenges, we present Random Projection-based Operator Networks (RandONets): shallow networks with random projections and tailor-made numerical analysis algorithms that learn linear and nonlinear operators. The implementation of RandONets involves: (a) incorporating random bases, thus enabling the use of shallow neural networks with a single hidden layer, where the only unknowns are the output weights of the network's weighted inner product; this reduces dramatically the dimensionality of the parameter space; and, based on this, (b) using tailor-made numerical analysis techniques to solve a linear ill-posed problem with regularization (e.g., Tikhonov regularization and preconditioned QR decomposition). In addition, we prove the universal approximation accuracy of RandONets for approximating linear and nonlinear operators. Furthermore, we demonstrate their efficiency in approximating linear and nonlinear evolution operators (right-hand-sides (RHS)) with a focus on PDEs. We also note that due to their simplicity, RandONets provide a one-step transformation of the input space, facilitating the interpretability. We show, that for this particular task, RandONets outperform both in terms of numerical approximation accuracy and computational cost, by several orders of magnitudes the "vanilla" DeepONets. Hence, we believe that our method will trigger further developments in the field of scientific machine learning, for the development of new ''light" machine learning schemes that will provide high accuracy while reducing dramatically the computational costs. A software/ toolbox in Matlab for RandONets with demos is also provided and available on GitHub at https://github.com/GianlucaFabiani/RandONets .

Training neural networks on high-dimensional data using random projection

Random Projection in Deep Neural Networks

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection.

Enhanced Expressive Power and Fast Training of Neural Networks by Random Projections

A survey of dimensionality reduction techniques based on random projection

Rademacher Random Projections with Tensor Networks

RandONets: Shallow Networks with Random Projections for Learning Linear and Nonlinear Operators

Nonparametric Locality Reforming Projection for Linear Discriminative Dimensionality Reduction

Mixed-Precision Random Projection for RandNLA on Tensor Cores

Sparse Data-Driven Random Projection in Regression for High-Dimensional Data

RandONet: Shallow-Networks with Random Projections for learning linear and nonlinear operators

Binary Random Projections with Controllable Sparsity Patterns

Hierarchical Feature Selection for Random Projection

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

High-dimensional clustering via Random Projections

Billion-scale Network Embedding with Iterative Random Projection

Tensor Random Projection for Low Memory Dimension Reduction

Random Polynomial Neural Networks: Analysis and Design

Nested Hyperbolic Spaces for Dimensionality Reduction and Hyperbolic NN Design

Random Projection-Based Dimensionality Reduction Method for Hyperspectral Target Detection