Abstract:The random weight idea in neural network has existed for long time and attracted wide attention for its faster learning speed than that of tuning-based methods. Recently, extreme learning machine (ELM), one non-tuning algorithm for single hidden layer neural network (SLFN) was proposed and applied in many applications. ELM can learn hundreds of times faster than that of the traditional tuning-based methods while obtaining better generalization performance. But since the ELM randomly assigns input weights, its stability and accuracy will decline sharply for high-dimension and small-sample dataset. Although dimension reduction can be used to improve it, this two-stage method will need more physical space and increase structure complexity of the model. In this paper, we propose a novel single-stage algorithm, base projection vector machine (BPVM) and its kernel version—kernel projection vector machine (KPVM) by combining dimension reduction and neural network training. The BPVM and KPVM are called projection vector machine (PVM) by a joint name. Different from random weight idea used in ELM, in PVM, the input weights of SLFN are obtained by singular decomposition, and the hidden neurons are ranked by their contribution level and only those that are important are selected. This will make PVM to obtain better generalization ability and more compact structure than ELM. Similar to ELM, PVM does not need any iterative steps, and thus can yield significantly fast training. Additionally, compared with the two-stage algorithms such as BP/SVD (BP co-work with singular value decomposition) and ELM/SVD (ELM co-work with singular value decomposition), PVM need less parameters and memory running space. The experimental results on many datasets show that; although PVM is not faster than ELM, it is much faster than BP, BP/SVD and ELM/SVD which produce the best generalization ability in most cases. Especially, PVM is very suitable for high-dimension and small-sample dataset.

VPNet: Variable Projection Networks

Weighted Hermite Variable Projection Networks for Classifying Visually Evoked Potentials

hvEEGNet: exploiting hierarchical VAEs on EEG data for neuroscience applications

Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE

vEEGNet: learning latent representations to reconstruct EEG raw data via variational autoencoders

An efficient and robust varying-parameter projection neural network for sparse signal reconstruction

D-VPnet: A Network for Real-time Dominant Vanishing Point Detection in Natural Scenes

Projection Vector Machine.

VS-Net: Variable splitting network for accelerated parallel MRI reconstruction

MS-NetVLAD: Multi-Scale NetVLAD for Visual Place Recognition

LVP-net: A deep network of learning visual pathway for edge detection

Whole-brain Prediction Analysis with GraphNet

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residual

TAVRNN: Temporal Attention-enhanced Variational Graph RNN Captures Neural Dynamics and Behavior

VPRTempo: A Fast Temporally Encoded Spiking Neural Network for Visual Place Recognition

Reliable Probabilistic Classification with Neural Networks

Random Projection in Deep Neural Networks

VC-PINN: Variable Coefficient Physical Information Neural Network For Forward And Inverse PDE Problems with Variable Coefficient

Efficient Scaling of Bayesian Neural Networks

Understanding Vector-Valued Neural Networks and Their Relationship with Real and Hypercomplex-Valued Neural Networks