Piecewise Linear Multilayer Perceptrons and Dropout

Ian J. Goodfellow

DOI: https://doi.org/10.48550/arXiv.1301.5088

IF: 5.414

2013-01-22

Machine Learning

Abstract:We propose a new type of hidden layer for a multilayer perceptron, and demonstrate that it obtains the best reported performance for an MLP on the MNIST dataset.

What problem does this paper attempt to address?

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Guangyong Chen,Pengfei Chen,Yujun Shi,Chang-Yu Hsieh,Benben Liao,Shengyu Zhang

DOI: https://doi.org/10.48550/arxiv.1905.05928

2019-01-01

Abstract:In this work, we propose a novel technique to boost training efficiency of a neural network. Our work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed. Given the well-known fact that independent components must be whitened, we introduce a novel Independent-Component (IC) layer before each weight layer, whose inputs would be made more independent. However, determining independent components is a computationally intensive task. To overcome this challenge, we propose to implement an IC layer by combining two popular techniques, Batch Normalization and Dropout, in a new manner that we can rigorously prove that Dropout can quadratically reduce the mutual information and linearly reduce the correlation between any pair of neurons with respect to the dropout layer parameter p. As demonstrated experimentally, the IC layer consistently outperforms the baseline approaches with more stable training process, faster convergence speed and better convergence limit on CIFAR10/100 and ILSVRC2012 datasets. The implementation of our IC layer makes us rethink the common practices in the design of neural networks. For example, we should not place Batch Normalization before ReLU since the non-negative responses of ReLU will make the weight layer updated in a suboptimal way, and we can achieve better performance by combining Batch Normalization and Dropout together as an IC layer.
Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

Taewoon Kim

DOI: https://doi.org/10.48550/arXiv.2108.08186

IF: 5.414

2021-08-18

Machine Learning

Abstract:A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g. faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/tae898/age-gender/
New Dynamical Optimal Learning for Linear Multilayer FNN

KC Tan,HJ Tang

DOI: https://doi.org/10.1109/tnn.2004.830801

2004-01-01

IEEE Transactions on Neural Networks

Abstract:This letter presents a new dynamical optimal learning (DOL) algorithm for three-layer linear neural networks and investigates its generalization ability. The optimal learning rates can be fully determined during the training process. The mean squared error (mse) is guaranteed to be stably decreased and the learning is less sensitive to initial parameter settings. The simulation results illustrate that the proposed DOL algorithm gives better generalization performance and faster convergence as compared to standard error back propagation algorithm.
Efficient Estimation of Multidimensional Regression Model using Multilayer Perceptrons

Joseph Rynkiewicz

DOI: https://doi.org/10.48550/arXiv.0802.3152

2008-02-22

Abstract:This work concerns the estimation of multidimensional nonlinear regression models using multilayer perceptrons (MLPs). The main problem with such models is that we need to know the covariance matrix of the noise to get an optimal estimator. However, we show in this paper that if we choose as the cost function the logarithm of the determinant of the empirical error covariance matrix, then we get an asymptotically optimal estimator. Moreover, under suitable assumptions, we show that this cost function leads to a very simple asymptotic law for testing the number of parameters of an identifiable MLP. Numerical experiments confirm the theoretical results.

Statistics Theory
Heterogeneous Multilayer Generalized Operational Perceptron

Dat Thanh Tran,Serkan Kiranyaz,Moncef Gabbouj,Alexandros Iosifidis

DOI: https://doi.org/10.1109/TNNLS.2019.2914082

2019-04-27

Abstract:The traditional Multilayer Perceptron (MLP) using McCulloch-Pitts neuron model is inherently limited to a set of neuronal activities, i.e., linear weighted sum followed by nonlinear thresholding step. Previously, Generalized Operational Perceptron (GOP) was proposed to extend conventional perceptron model by defining a diverse set of neuronal activities to imitate a generalized model of biological neurons. Together with GOP, Progressive Operational Perceptron (POP) algorithm was proposed to optimize a pre-defined template of multiple homogeneous layers in a layerwise manner. In this paper, we propose an efficient algorithm to learn a compact, fully heterogeneous multilayer network that allows each individual neuron, regardless of the layer, to have distinct characteristics. Based on the complexity of the problem, the proposed algorithm operates in a progressive manner on a neuronal level, searching for a compact topology, not only in terms of depth but also width, i.e., the number of neurons in each layer. The proposed algorithm is shown to outperform other related learning methods in extensive experiments on several classification problems.

Neural and Evolutionary Computing
A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

Xinjie Lan,Kenneth E. Barner

DOI: https://doi.org/10.48550/arXiv.2010.14054

2020-10-27

Abstract:In this paper, we propose a probabilistic representation of MultiLayer Perceptrons (MLPs) to improve the information-theoretic interpretability. Above all, we demonstrate that the activations being i.i.d. is not valid for all the hidden layers of MLPs, thus the existing mutual information estimators based on non-parametric inference methods, e.g., empirical distributions and Kernel Density Estimate (KDE), are invalid for measuring the information flow in MLPs. Moreover, we introduce explicit probabilistic explanations for MLPs: (i) we define the probability space (Omega_F, t, P_F) for a fully connected layer f and demonstrate the great effect of an activation function on the probability measure P_F ; (ii) we prove the entire architecture of MLPs as a Gibbs distribution P; and (iii) the back-propagation aims to optimize the sample space Omega_F of all the fully connected layers of MLPs for learning an optimal Gibbs distribution P* to express the statistical connection between the input and the label. Based on the probabilistic explanations for MLPs, we improve the information-theoretic interpretability of MLPs in three aspects: (i) the random variable of f is discrete and the corresponding entropy is finite; (ii) the information bottleneck theory cannot correctly explain the information flow in MLPs if we take into account the back-propagation; and (iii) we propose novel information-theoretic explanations for the generalization of MLPs. Finally, we demonstrate the proposed probabilistic representation and information-theoretic explanations for MLPs in a synthetic dataset and benchmark datasets.

Machine Learning,Information Theory
Learning multiple layers of representation

Geoffrey E Hinton

DOI: https://doi.org/10.1016/j.tics.2007.09.004

Abstract:To achieve its impressive performance in tasks such as speech perception or object recognition, the brain extracts multiple levels of representation from the sensory input. Backpropagation was the first computationally efficient model of how neural networks could learn multiple layers of representation, but it required labeled training data and it did not work well in deep networks. The limitations of backpropagation learning can now be overcome by using multilayer neural networks that contain top-down connections and training them to generate sensory data rather than to classify it. Learning multilayer generative models might seem difficult, but a recent discovery makes it easy to learn nonlinear distributed representations one layer at a time.
Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Ke-Lin Du,Chi-Sing Leung,M. N. S. Swamy,Wai-Ho Mow

DOI: https://doi.org/10.3390/math10244730

IF: 2.4

2022-12-14

Mathematics

Abstract:The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

mathematics
Parameter efficient dendritic-tree neurons outperform perceptrons

Ziwen Han,Evgeniya Gorobets,Pan Chen

DOI: https://doi.org/10.48550/arXiv.2207.00708

2022-07-02

Neural and Evolutionary Computing

Abstract:Biological neurons are more powerful than artificial perceptrons, in part due to complex dendritic input computations. Inspired to empower the perceptron with biologically inspired features, we explore the effect of adding and tuning input branching factors along with input dropout. This allows for parameter efficient non-linear input architectures to be discovered and benchmarked. Furthermore, we present a PyTorch module to replace multi-layer perceptron layers in existing architectures. Our initial experiments on MNIST classification demonstrate the accuracy and generalization improvement of dendritic neurons compared to existing perceptron architectures.
Deep Learning under Privileged Information Using Heteroscedastic Dropout

John Lambert,Ozan Sener,Silvio Savarese

DOI: https://doi.org/10.48550/arXiv.1805.11614

2018-05-30

Abstract:Unlike machines, humans learn through rapid, abstract model-building. The role of a teacher is not simply to hammer home right or wrong answers, but rather to provide intuitive comments, comparisons, and explanations to a pupil. This is what the Learning Under Privileged Information (LUPI) paradigm endeavors to model by utilizing extra knowledge only available during training. We propose a new LUPI algorithm specifically designed for Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). We propose to use a heteroscedastic dropout (i.e. dropout with a varying variance) and make the variance of the dropout a function of privileged information. Intuitively, this corresponds to using the privileged information to control the uncertainty of the model output. We perform experiments using CNNs and RNNs for the tasks of image classification and machine translation. Our method significantly increases the sample efficiency during learning, resulting in higher accuracy with a large margin when the number of training examples is limited. We also theoretically justify the gains in sample efficiency by providing a generalization error bound decreasing with $O(\frac{1}{n})$, where $n$ is the number of training examples, in an oracle case.

Machine Learning
Efficient Estimation of Multidimensional Regression Model with Multilayer Perceptron

Joseph Rynkiewicz

DOI: https://doi.org/10.48550/arXiv.0802.3142

2008-02-21

Statistics Theory

Abstract:This work concerns estimation of multidimensional nonlinear regression models using multilayer perceptron (MLP). The main problem with such model is that we have to know the covariance matrix of the noise to get optimal estimator. however we show that, if we choose as cost function the logarithm of the determinant of the empirical error covariance matrix, we get an asymptotically optimal estimator.
A Modular Single-Hidden-Layer Perceptron For Letter Recognition

Dq Gao,Sm Zhu,Wb Gu

DOI: https://doi.org/10.1007/11550822_72

2005-01-01

Abstract:An n-class problem is decomposed into n two-class problems. Naturally, modular multilayer perceptions (MLPs) come into being. A single-output MLP is behalf of a class and trained by a two-class learning subset. A training subset only consists of all samples from a special class and a part samples from the nearest classes. If the decision boundary of a single-output MLP is open, its outputs are amended by a correction coefficient. This paper clarifies such a fact that the generalization of a single-output MLP is seriously affected by the sample disequilibrium situation. Therefore, the samples from the little class have to be multiplied an enlarging factor. The result of letter recognition shows that the above methods are effective.
Dropout, a basic and effective regularization method for a deep learning model: a case study

Brahim Jabir,Noureddine Falih

DOI: https://doi.org/10.11591/ijeecs.v24.i2.pp1009-1016

2021-11-01

Indonesian Journal of Electrical Engineering and Computer Science

Abstract:Deep learning is based on a network of artificial neurons inspired by the human brain. This network is made up of tens or even hundreds of "layers" of neurons. The fields of application of deep learning are indeed multiple; Agriculture is one of those fields in which deep learning is used in various agricultural problems (disease detection, pest detection, and weed identification). A major problem with deep learning is how to create a model that works well, not only on the learning set but also on the validation set. Many approaches used in neural networks are explicitly designed to reduce overfit, possibly at the expense of increasing validation accuracy and training accuracy. In this paper, a basic technique (dropout) is proposed to minimize overfit, we integrated it into a convolutional neural network model to classify weed species and see how it impacts performance, a complementary solution (exponential linear units) are proposed to optimize the obtained results. The results showed that these proposed solutions are practical and highly accurate, enabling us to adopt them in deep learning models.
Fast Learning Network with Parallel Layer Perceptrons

Guoqiang Li,Xiaobin Qi,Bin Chen,Yunpeng Ma,Peifeng Niu,Zhiwang Chen

DOI: https://doi.org/10.1007/s11063-017-9667-6

IF: 2.565

2017-01-01

Neural Processing Letters

Abstract:This paper proposes a novel artificial neural network called Parallel Layer Perceptron Fast Learning Network (PLP-FLN). In PLP-FLN, a parallel single hidden layer feed-forward neural network is added on the basis of Fast Learning Network (FLN) which is an improved Extreme Learning Machine (ELM). Input weights and hidden layer biases are randomly generated. The weights connect the output nodes and the input nodes, and the weights connect the output nodes and the hidden nodes are analytically determined based on least squares methods. In order to test the PLP-FLN validity, this paper compared it with ELM, FLN, Kernel ELM and Incremental ELM through 12 regression applications and 7 classification problems. By comparing the experimental results, it shows that the PLP-FLN with much more compact networks have demonstrated better approximations, classification performances and generalization ability.
Layer-wise Regularized Dropout for Neural Language Models

Shiwen Ni,Min Yang,Ruifeng Xu,Chengming Li,Xiping Hu

2024-02-26

Abstract:Among the various pre-trained neural language models that are popular today, dropout is already an indispensable regularization technique. To solve the inconsistency between training and inference caused by the randomness of dropout, some studies use consistency training to regularize dropout at the output layer. In this paper, we propose a novel Layer-wise Regularized Dropout (LR-Drop), which is specially designed for Transformer-based Language models. Specifically, LR-Drop layer-wise regularizes each Transformer layer using the consistency training strategy. Each training sample passes through the two siamese sub-models sampled by dropout, and then LR-Drop forces the hidden states, multi-head attention matrices, and output distribution of the two siamese sub-models to be consistent. The proposed LR-Drop can be regarded as a "self-distillation" framework, in which each sub-model generated by dropout is the other's "teacher" model and "student" model. Through extensive experiments on 8 natural language understanding datasets, 6 neural machine translation datasets, and 1 abstractive summarization dataset (a total of 15 datasets), we show that LR-Drop achieves superior performances, including state-of-the-art results.

Artificial Intelligence
BCMLP: Binary-connected multilayer perceptrons

Luo Ningqi,Song Binheng,Pan Yinxu,Shen Bin

DOI: https://doi.org/10.1007/978-3-030-04179-3_7

2018-01-01

Abstract:Sparse connection has been used both to reduce network complexity and sensitivity with input perturbations in multilayer perceptrons as well as artificial neural networks. We propose a novel binary-connected multilayer perceptrons where arbitrary node is connected with the only two nodes of previous layer. The sensitivity of this model is discussed both in theoretical methods and simulation experiments. Comparisons with related works show that our scheme achieves the least amount of parameters, the lowest deviation to input perturbations, and the highest accuracy in the noisy classification task.
Guided Dropout: Improving Deep Networks Without Increased Computation

Yifeng Liu,Yangyang Li,Zhongxiong Xu,Xiaohan Liu,Haiyong Xie,Huacheng Zeng

DOI: https://doi.org/10.32604/iasc.2023.033286

2023-01-01

Intelligent Automation & Soft Computing

Abstract:Deep convolution neural networks are going deeper and deeper. However, the complexity of models is prone to overfitting in training. Dropout, one of the crucial tricks, prevents units from co-adapting too much by randomly dropping neurons during training. It effectively improves the performance of deep networks but ignores the importance of the differences between neurons. To optimize this issue, this paper presents a new dropout method called guided dropout, which selects the neurons to switch off according to the differences between the convolution kernel and preserves the informative neurons. It uses an unsupervised clustering algorithm to cluster similar neurons in each hidden layer, and dropout uses a certain probability within each cluster. Thereby this would preserve the hidden layer neurons with different roles while maintaining the model's scarcity and generalization, which effectively improves the role of the hidden layer neurons in learning the features. We evaluated our approach compared with two standard dropout networks on three well-established public object detection datasets. Experimental results on multiple datasets show that the method proposed in this paper has been improved on false positives, precision-recall curve and average precision without increasing the amount of computation. It can be seen that the increased performance of guided dropout is thanks to shallow learning in the networks. The concept of guided dropout would be beneficial to the other vision tasks.
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal,Zoubin Ghahramani

DOI: https://doi.org/10.48550/arXiv.1512.05287

2016-10-05

Abstract:Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

Machine Learning
A Smooth Optimisation Perspective on Designing and Training Feedforward Multilayer Perceptrons

Hao Shen

2016-01-01

Abstract:Despite the recent success of deep neural networks in various applications, designing and training deep neural networks is still among the greatest challenges in the field. In this work, we address the challenge of designing and training feedforward Multilayer Perceptrons (MLPs) from a smooth optimisation perspective. By characterising the critical point conditions of an MLP based loss function, we identify conditions to eliminate local optima of the corresponding cost function. By studying the Hessian structure of the cost function at the global minima, we develop an approximate Newtonu0027s MLP algorithm. Our results are demonstrated on an analysis of MLPs with only one hidden layer, and numerically evaluated on the benchmark problem of four region classification.
Point Cloud Network: An Order of Magnitude Improvement in Linear Layer Parameter Count

Charles Hetterich

2023-09-23

Abstract:This paper introduces the Point Cloud Network (PCN) architecture, a novel implementation of linear layers in deep learning networks, and provides empirical evidence to advocate for its preference over the Multilayer Perceptron (MLP) in linear layers. We train several models, including the original AlexNet, using both MLP and PCN architectures for direct comparison of linear layers (Krizhevsky et al., 2012). The key results collected are model parameter count and top-1 test accuracy over the CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009). AlexNet-PCN16, our PCN equivalent to AlexNet, achieves comparable efficacy (test accuracy) to the original architecture with a 99.5% reduction of parameters in its linear layers. All training is done on cloud RTX 4090 GPUs, leveraging pytorch for model construction and training. Code is provided for anyone to reproduce the trials from this paper.

Machine Learning,Computer Vision and Pattern Recognition,Neural and Evolutionary Computing

Piecewise Linear Multilayer Perceptrons and Dropout

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

New Dynamical Optimal Learning for Linear Multilayer FNN

Efficient Estimation of Multidimensional Regression Model using Multilayer Perceptrons

Heterogeneous Multilayer Generalized Operational Perceptron

A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

Learning multiple layers of representation

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Parameter efficient dendritic-tree neurons outperform perceptrons

Deep Learning under Privileged Information Using Heteroscedastic Dropout

Efficient Estimation of Multidimensional Regression Model with Multilayer Perceptron

A Modular Single-Hidden-Layer Perceptron For Letter Recognition

Dropout, a basic and effective regularization method for a deep learning model: a case study

Fast Learning Network with Parallel Layer Perceptrons

Layer-wise Regularized Dropout for Neural Language Models

BCMLP: Binary-connected multilayer perceptrons

Guided Dropout: Improving Deep Networks Without Increased Computation

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

A Smooth Optimisation Perspective on Designing and Training Feedforward Multilayer Perceptrons

Point Cloud Network: An Order of Magnitude Improvement in Linear Layer Parameter Count