Abstract:Neural networks, as powerful tools for data mining and knowledge engineering, can learn from data to build feature‐based classifiers and nonlinear predictive models. Training neural networks involves the optimization of nonconvex objective functions, and usually, the learning process is costly and infeasible for applications associated with data streams. A possible, albeit counterintuitive, alternative is to randomly assign a subset of the networks’ weights so that the resulting optimization task can be formulated as a linear least‐squares problem. This methodology can be applied to both feedforward and recurrent networks, and similar techniques can be used to approximate kernel functions. Many experimental results indicate that such randomized models can reach sound performance compared to fully adaptable ones, with a number of favorable benefits, including (1) simplicity of implementation, (2) faster learning with less intervention from human beings, and (3) possibility of leveraging overall linear regression and classification algorithms (e.g., ℓ 1 norm minimization for obtaining sparse formulations). This class of neural networks attractive and valuable to the data mining community, particularly for handling large scale data mining in real‐time. However, the literature in the field is extremely vast and fragmented, with many results being reintroduced multiple times under different names. This overview aims to provide a self‐contained, uniform introduction to the different ways in which randomization can be applied to the design of neural networks and kernel functions. A clear exposition of the basic framework underlying all these approaches helps to clarify innovative lines of research, open problems, and most importantly, foster the exchanges of well‐known results throughout different communities. WIREs Data Mining Knowl Discov 2017, 7:e1200. doi: 10.1002/widm.1200This article is categorized under: Technologies > Machine Learning

Feasibility of random basis function approximators for modeling and control

Universal approximation property of Banach space-valued random feature models including random neural networks

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Random Vector Functional Link Networks for Function Approximation on Manifolds

Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control

Random Features Approximation for Control-Affine Systems

Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

On the Approximation Properties of Random ReLU Features

Insights into Randomized Algorithms for Neural Networks: Practical Issues and Common Pitfalls.

Understanding Approximation for Bayesian Inference in Neural Networks

Approximation of RKHS Functionals by Neural Networks

Basis Operator Network: A Neural Network-Based Model for Learning Nonlinear Operators Via Neural Basis

Random Fourier Approximation of the Kernel Function in Programmable Networks

Why Deep Neural Networks for Function Approximation?

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Function Gradient Approximation with Random Shallow ReLU Networks with Control Applications

Randomness in Neural Networks: an Overview

Universal Approximation Using Radial-Basis-Function Networks

Investigation of Neural Networks for Function Approximation

Multi-layer random features and the approximation power of neural networks