Abstract:Recent work has focused on data-driven learning of the evolution of unknown systems via deep neural networks (DNNs), with the goal of conducting long time prediction of the evolution of the unknown system. Training a DNN with low generalization error is a particularly important task in this case as error is accumulated over time. Because of the inherent randomness in DNN training, chiefly in stochastic optimization, there is uncertainty in the resulting prediction, and therefore in the generalization error. Hence, the generalization error can be viewed as a random variable with some probability distribution. Well-trained DNNs, particularly those with many hyperparameters, typically result in probability distributions for generalization error with low bias but high variance. High variance causes variability and unpredictably in the results of a trained DNN. This paper presents a computational technique which decreases the variance of the generalization error, thereby improving the reliability of the DNN model to generalize consistently. In the proposed ensemble averaging method, multiple models are independently trained and model predictions are averaged at each time step. A mathematical foundation for the method is presented, including results regarding the distribution of the local truncation error. In addition, three time-dependent differential equation problems are considered as numerical examples, demonstrating the effectiveness of the method to decrease variance of DNN predictions generally.

Random Machines Regression Approach: an ensemble support vector regression model with free kernel choice

Support Vector Machines Ensemble With Optimizing Weights By Genetic Algorithm

Genetic Ensemble of Extreme Learning Machine

Dynamically Weighted Ensemble Neural Networks for Regression Problems

Bagging Support Vector Data Description Model for Batch Process Monitoring

Randomized Kernel Methods for Least-Squares Support Vector Machines

Fuzzy Support Vector Machine For Regression Estimation

A practical tutorial on bagging and boosting based ensembles for machine learning: Algorithms, software tools, performance study, practical perspectives and opportunities

Cost-sensitive probabilistic predictions for support vector machines

Support Vector Regression Machines

RANDOM FOREST AND SUPPORT VECTOR MACHINE ON FEATURES SELECTION FOR REGRESSION ANALYSIS

Random Bits Regression: a Strong General Predictor for Big Data

Boosting Support Vector Machines Successfully.

Supply chain finance credit risk assessment using support vector machine–based ensemble improved with noise elimination

Towards a Systematic Approach to Design New Ensemble Learning Algorithms

Ensembles of Probabilistic Regression Trees

Systematic Ensemble Learning for Regression

Designing Ensemble Learning Algorithms Using Kernel Methods

Machine Learning for Multi-Output Regression: When should a holistic multivariate approach be preferred over separate univariate ones?

A Learnable Kernel Machine for Short-Term Load Forecasting

Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning