Ensembles of Surrogate Models

Ping Jiang,Qi Zhou,Xinyu Shao

DOI: https://doi.org/10.1007/978-981-15-0731-1_3

2020-01-01

Abstract:An ensemble of surrogate models (EM) is a surrogate model composed of a series of surrogate models combined through a weighted sum. An EM can take advantage of each individual surrogate model to effectively increase the robustness of the prediction. The mathematical expression for an EM can be given as follows:

What problem does this paper attempt to address?

Support Vector Machines Ensemble With Optimizing Weights By Genetic Algorithm

Ling-Min He,Xiao-Bing Yang,Fan-Sheng Kong

DOI: https://doi.org/10.1109/ICMLC.2006.258541

2006-01-01

Abstract:Support Vector Machines (SVM) is a classification technique based on the structural risk minimization principle. It is characteristic of processing complex data and high accuracy. And the ensemble of classifiers often has better performance than any of component classifiers in the ensemble. In this paper, bagging, boosting, multiple SVM decision model (MSDM) and heterogeneous SVM decision model (HSDM) of SVM ensemble are compared on four data sets. For boosting and bagging, genetic algorithm is used to optimize the combining weights of component SVMs. Experiment results show that SVM ensemble with optimizing weights by genetic algorithm could gain higher accuracy.
Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

Zhang-quan Shen,Fan-sheng Kong

DOI: https://doi.org/10.1007/978-3-540-28647-9_55

2004-01-01

Abstract:Combining the outputs of several neural networks into an aggregate output often gives improved accuracy over any individual output. The set of networks is known as an ensemble. Neural network ensembles are effective techniques to improve the generalization of a neural network system. This paper presents an ensemble method for regression that has advantages over simple weighted or weighted average combining techniques. After the training of component neural networks, genetic algorithm is used to optimize the combining weights of component networks. Compared with ordinary weighted methods, the method proposed in this paper achieved high predicting accuracy on five test datasets.
MID-ESM: An Adaptive Median-Based Ensemble of Surrogate Models

Jianji Li,Shuai Zhang,Pengwei Liang,Xiaonan Lai,Xueguan Song

DOI: https://doi.org/10.1115/1.4062977

2023-07-18

Journal of Mechanical Design

Abstract:Abstract Along with the development of surrogate models, there is a growing need to use surrogate models instead of computationally intensive simulations to estimate real system responses. Compared with individual surrogate models, the ensemble of surrogate models is gradually drawing more attention due to its better applicability and robustness. Thus, this paper proposes an adaptive median-based ensemble of surrogate models (MID-ESM). At first, construct a reference model using the median of the predicted values of several surrogate models. Then an adaptive weight ensemble strategy is proposed based on the reference model to integrate global trends and local features. Thirty test functions and a practical engineering case are used to evaluate the model performance. In addition, this paper investigates the effect of homoscedasticity noise and test functions of different dimensions on the proposed model. The results demonstrate that MID-ESM has higher accuracy and robustness than individual surrogate models and other ensembles of surrogate models, offering better applicability in engineering problems.

engineering, mechanical
Deep Residual Surrogate Model

Tianxin Huang,Yong Liu,Zaisheng Pan

DOI: https://doi.org/10.1016/j.ins.2022.04.041

IF: 8.1

2022-01-01

Information Sciences

Abstract:Surrogate models are widely used to model the high computational cost problems such as industrial simulation or engineering optimization when the size of sampled data for modeling is greatly limited. They can significantly improve the efficiency of complex calculations by modeling original expensive problems with simpler computation-saving functions. However, a single surrogate model cannot always perform well for various problems. On this occasion, hybrid surrogate models are created to improve the final performances on different problems by combining advantages of multiple single models. Nevertheless, existing hybrid methods work by estimating weights for all alternative single models, which limits the efficiency when more single models are adopted. In this paper, we propose a novel hybrid surrogate model quite different from former methods, named the Deep Residual Surrogate model (DRS). DRS does not merge all alternative single surrogate models directly by weights, but by assembling selected ones in a multiple layers structure. We propose first derivate validation (FDV) to recurrently select the single surrogate model adopted in each layer from all candidates. Experimental results on multiple benchmark problems demonstrate that DRS has better performances than existing single and hybrid surrogate models in both prediction accuracy and stability with higher efficiency.
Ensemble of surrogates in black-box-type engineering optimization: Recent advances and applications

Hao Chen,Zhilang Zhang,Weikun Li,Qimeng Liu,Kai Sun,Dixia Fan,Weicheng Cui

DOI: https://doi.org/10.1016/j.eswa.2024.123427

IF: 8.5

2024-02-16

Expert Systems with Applications

Abstract:Due to its high efficiency, surrogate models have been extensively used in black-box-type engineering optimization problems. However, due to the nature of black-box functions, it is difficult to decide which surrogate model is the best for a given problem. In addition, it is difficult to accurately estimate the prediction accuracy of a surrogate on an expensive black-box function in advance. Because we cannot use many real expensive samples to test the prediction accuracy of different surrogates due to the limited computational resources. Ensembles of surrogates fully exploit the advantages of each surrogate model in predicting a given implicit function and have been shown to be effective in improving the robustness of surrogates. The use of ensembles of surrogates to the engineering design optimization has recently attracted huge attention. This work reviews the state-of-the-art ensembles of surrogates and their applications in black-box-type engineering optimization problems. We first familiarize readers with ensembles of surrogates, and give a description of different types of ensembles of surrogates. Further, the recent advances in using ensembles of surrogates for black-box-type engineering optimization are reviewed, and the applications in various optimization needs are highlighted. Finally, we identify the research gaps and most important trends. This review aims to serve as a comprehensive guide that enhances the accessibility of ensembles of surrogates, which can act as an insurance policy in predicting expensive black-box functions and are increasingly used in the field of black-box-type engineering optimization.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy

Peng Zhang,Shuyou Zhang,Xiaojian Liu,Lemiao Qiu,Guodong Yi

DOI: https://doi.org/10.3390/app9091845

2019-01-01

Abstract:Surrogate models are often used as alternatives to considerably reduce the computational burden of the expensive computer simulations that are required for engineering designs. The development of surrogate models for complex relationships between the parameters often requires the modeling of high-dimensional functions with limited information, and it is challenging to choose an effective surrogate model over the unknown design space. To this end, the ensemble modelscombined with different surrogate modelsoffer effective solutions. This paper presents a new ensemble model based on the least squares method, which is a regularization strategy and an augmentation strategy; we call the model the regularized least squares ensemble model (RLS-EM). Three individual surrogate modelsKriging, radial basis function, and support vector regressionare used to compose the RLS-EM. Further, the weight factors are estimated by the least squares method without using the global or local error metrics, which are used in most existing methods. To solve the collinearity in the least squares calculation process, a regularization strategy and an augmentation strategy are developed. The two strategies help explore the unknown regions and improve the accuracy on one hand; on the other hand, the collinearity can be reduced, and the overfitting phenomenon that may occur can be avoided. Six numerical functions, from two-dimensional to 12-dimensional, and a computer numerical control (CNC) milling machine bed design problem are used to verify the proposed method. The results of the numerical examples show that RLS-EM saves a considerable amount of computation time while ensuring the same level of robustness and accuracy compared with other ensemble models. The RLS-EM used for the CNC milling machine bed design problem also shows good accuracy characteristics compared with other ensemble methods.
The Ensemble of Surrogate Model Based on Local and Global Errors

X Zhang,H Li,G Xiang,H W Xu

DOI: https://doi.org/10.1088/1757-899x/1043/5/052049

2021-01-01

IOP Conference Series Materials Science and Engineering

Abstract:Abstract The weight coefficient is the key factor for the success of the ensemble of surrogate model construction. In this paper, a method for constructing ensemble of surrogate model by combining local and global errors to calculate weight coefficient has been put forward. Radial basis function (RBF) and the Kriging model are built as the meta models, and the cross validation strategy is applied to calculate the global errors and the local errors of samples. The inverse proportion average method is used to calculate the weight coefficient by combining the local errors and global errors. In order to verify the effectiveness of the proposed method, two meta models and three methods to construct the ensemble of surrogate models are tested with six benchmark functions. The results show that the proposed method can improve the accuracy, robustness and universality of the surrogate model.
A Study on Metamodeling Techniques, Ensembles, and Multi-Surrogates in Evolutionary Computation

Dudy Lim,Yew-Soon Ong,Yaochu Jin,Bernhard Sendhoff

DOI: https://doi.org/10.1145/1276958.1277203

2007-01-01

Abstract:Surrogate-Assisted Memetic Algorithm (SAMA) is a hybrid evolutionary algorithm, particularly a memetic algorithm that employs surrogate models in the optimization search. Since most of the objective function evaluations in SAMA are approximated, the search performance of SAMA is likely to be affected by the characteristics of the models used. In this paper, we study the search performance of using different meta modeling techniques, ensembles, and multi-surrogates in SAMA. In particular, we consider the SAMA-TRF, a SAMA model management framework that incorporates a trust region scheme for interleaving use of exact objective function with computationally cheap local meta models during local searches. Four different metamodels, namely Gaussian Process (GP), Radial Basis Function (RBF), Polynomial Regression (PR), and Extreme Learning Machine (ELM) neural network are used in the study. Empirical results obtained show that while some metamodeling techniques perform best on particular benchmark problems, ensemble of metamodels and multisurrogates yield robust and improved solution quality on the benchmark problems in general, for the same computational budget.
An Evolution Strategy Assisted by an Ensemble of Local Gaussian Process Models.

Jianfeng Lu,Bin Li,Yaochu Jin

DOI: https://doi.org/10.1145/2463372.2463425

2013-01-01

Abstract:Surrogate models used in evolutionary algorithms (EAs) aim to reduce computationally expensive objective function evaluations. However, low-quality surrogates may mislead EAs and as a result, surrogate-assisted EAs may fail to locate the global optimum. Among various machine learning models for surrogates, Gaussian Process (GP) models have shown to be effective as GP models are able to provide fitness estimation as well as a confidence level. One weakness of GP models is that the computational cost for training increases rapidly as the number of training samples increases. To reduce the computational cost for training, here we propose to adopt an ensemble of local Gaussian Process models. Different from independent local Gaussian Process models, local Gaussian Process models share the same model parameters. Then the performance of the covariance matrix adaptation evolution strategy (CMA-ES) assisted by an ensemble of local Gaussian Process models with five different sampling strategies is compared. Experiments on eight benchmark functions demonstrate that ensembles of local Gaussian Process models can provide reliable fitness prediction and uncertainty estimation. Among the compared strategies, the clustering technique using the lower confidence bound sampling strategy exhibits the best global search performance.
A surrogate-ensemble assisted expensive many-objective optimization

Yi Zhao,Chaoli Sun,Jianchao Zeng,Ying Tan,Guochen Zhang

DOI: https://doi.org/10.1016/j.knosys.2020.106520

2021-01-01

Abstract:<p>The surrogate ensemble has been shown to have a good performance for assisting the evolutionary algorithms to solve computationally expensive single-objective problems. However, it has not been paid much attention to assist in multi-/many-objective optimization. In this paper, we propose to train multiple surrogate models to assist many-objective optimization algorithm for solving expensive many-objective problems. In the proposed method, some non-dominated solutions, which have been evaluated using the expensive objectives, will be utilized together with the current population to generate the offspring in order to prevent the population from moving forward to a bad region. In the model management, the way to determine the method used for selecting individuals to be evaluated using the exact expensive objectives depends on the convergence degree of the current population. A new method to measure the uncertainty is also proposed in our method by considering the distance to the samples in the decision space and the approximation variance in the objective space. The experimental results on a number of benchmark problems show that our proposed method is competitive to two state-of-the-art surrogate-assisted evolutionary algorithms for solving expensive many-objective problems in a limited computational budget.</p>

computer science, artificial intelligence
A Study on Metamodeling Techniques, Ensembles, and Multi-Surrogates in Surrogate-Assisted Memetic Algorithms

Dudy Lim,Yew-Soon Ong,Yaochu Jin,Bernhard Sendhoff

2007-01-01

Abstract:Surrogate-Assisted Memetic Algorithm(SAMA) is a hybrid evolutionary algorithm, particularly a memetic algorithm that employs surrogate models in the optimization search. Since most of the objective function evaluations in SAMA are approximated, the search performance of SAMA is likely to be afiected by the characteristics of the models used. In this paper, we study the search performance of using difierent metamodeling techniques, ensembles, and multi- surrogates in SAMA. In particular, we consider the SAMA- TRF, a SAMA model management framework that incorpo- rates a trust region scheme for interleaving use of exact ob- jective function with computationally cheap local metamod- els during local searches. Four difierent metamodels, namely Gaussian Process (GP), Radial Basis Function (RBF), Poly- nomial Regression (PR), and Extreme Learning Machine (ELM) neural network are used in the study. Empirical results obtained show that while some metamodeling tech- niques perform best on particular benchmark problems, en- semble of metamodels and multi-surrogates yield robust and improved solution quality on the benchmark problems in general, for the same computational budget.
hubEnsembles: Ensembling Methods in R

Li Shandross,Emily Howerton,Lucie Contamin,Harry Hochheiser,Anna Krystalli,Consortium of Infectious Disease Modeling Hubs,Nicholas G. Reich,Evan L. Ray

DOI: https://doi.org/10.1101/2024.06.24.24309416

2024-06-25

Abstract:Combining predictions from multiple models into an ensemble is a widely used practice across many fields with demonstrated performance benefits. The R package provides a flexible framework for ensembling various types of predictions, including point estimates and probabilistic predictions. A range of common methods for generating ensembles are supported, including weighted averages, quantile averages, and linear pools. The package fits within a broader framework of open-source software and data tools called the “hubverse”, which facilitates the development and management of collaborative modelling exercises.

Public and Global Health
Synergy-of-Experts: Collaborate to Improve Adversarial Robustness

Sen Cui,Jingfeng Zhang,Jian Liang,Bo Han,Masashi Sugiyama,Changshui Zhang

2022-01-01

Abstract:Learning adversarially robust models require invariant predictions to a small neighborhood of its natural inputs, often encountering insufficient model capacity. There is research showing that learning multiple sub-models in an ensemble could mitigate this insufficiency, further improving the generalization and the robustness. However, the ensemble's voting-based strategy excludes the possibility that the true predictions remain with the minority. Therefore, this paper further improves the ensemble through a collaboration scheme---Synergy-of-Experts (SoE). Compared with the voting-based strategy, the SoE enables the possibility of correct predictions even if there exists a single correct sub-model. In SoE, every sub-model fits its specific vulnerability area and reserves the rest of the sub-models to fit other vulnerability areas, which effectively optimizes the utilization of the model capacity. Empirical experiments verify that SoE outperforms various ensemble methods against white-box and transfer-based adversarial attacks.
An Ensemble Surrogate-based Framework for Expensive Multiobjective Evolutionary Optimization

Qiuzhen Lin,Xunfeng Wu,Lijia Ma,Jianqiang Li,Maoguo Gong,Carlos A. Coello Coello

DOI: https://doi.org/10.1109/tevc.2021.3103936

IF: 16.497

2021-01-01

IEEE Transactions on Evolutionary Computation

Abstract:Surrogate-assisted evolutionary algorithms (SAEAs) have become very popular for tackling computationally expensive multiobjective optimization problems (EMOPs), as the surrogate models in SAEAs can approximate EMOPs well, thereby reducing the time cost of the optimization process. However, with the increased number of decision variables in EMOPs, the prediction accuracy of surrogate models will deteriorate, which inevitably worsens the performance of SAEAs. To deal with this issue, this article suggests an ensemble surrogate-based framework for tackling EMOPs. In this framework, a global surrogate model is trained under the entire search space to explore the global area, while a number of surrogate submodels are trained under different search subspaces to exploit the subarea, so as to enhance the prediction accuracy and reliability. Moreover, a new infill sampling criterion is designed based on a set of reference vectors to select promising samples for training the models. To validate the generality and effectiveness of our framework, three state-of-the-art evolutionary algorithms [nondominated sorting genetic algorithm III (NSGA-III), multiobjective evolutionary algorithm based on decomposition with differential evolution (MOEA/D-DE) and reference vector-guided evolutionary algorithm (RVEA)] are embedded, which significantly improve their performance for solving most of the test EMOPs adopted in this article. When compared to some competitive SAEAs for solving EMOPs with up to 30 decision variables, the experimental results also validate the advantages of our approach in most cases.

computer science, artificial intelligence, theory & methods
Are Ensembles Getting Better all the Time?

Pierre-Alexandre Mattei,Damien Garreau

2024-03-20

Abstract:Ensemble methods combine the predictions of several base models. We study whether or not including more models always improves their average performance. This question depends on the kind of ensemble considered, as well as the predictive metric chosen. We focus on situations where all members of the ensemble are a priori expected to perform as well, which is the case of several popular methods such as random forests or deep ensembles. In this setting, we show that ensembles are getting better all the time if, and only if, the considered loss function is convex. More precisely, in that case, the average loss of the ensemble is a decreasing function of the number of models. When the loss function is nonconvex, we show a series of results that can be summarised as: ensembles of good models keep getting better, and ensembles of bad models keep getting worse. To this end, we prove a new result on the monotonicity of tail probabilities that may be of independent interest. We illustrate our results on a medical prediction problem (diagnosing melanomas using neural nets) and a "wisdom of crowds" experiment (guessing the ratings of upcoming movies).

Machine Learning,Statistics Theory,Methodology
Deep interpretable ensembles

Lucas Kook,Andrea Götschi,Philipp FM Baumann,Torsten Hothorn,Beate Sick

DOI: https://doi.org/10.48550/arXiv.2205.12729

IF: 5.414

2022-05-25

Machine Learning

Abstract:Ensembles improve prediction performance and allow uncertainty quantification by aggregating predictions from multiple models. In deep ensembling, the individual models are usually black box neural networks, or recently, partially interpretable semi-structured deep transformation models. However, interpretability of the ensemble members is generally lost upon aggregation. This is a crucial drawback of deep ensembles in high-stake decision fields, in which interpretable models are desired. We propose a novel transformation ensemble which aggregates probabilistic predictions with the guarantee to preserve interpretability and yield uniformly better predictions than the ensemble members on average. Transformation ensembles are tailored towards interpretable deep transformation models but are applicable to a wider range of probabilistic neural networks. In experiments on several publicly available data sets, we demonstrate that transformation ensembles perform on par with classical deep ensembles in terms of prediction performance, discrimination, and calibration. In addition, we demonstrate how transformation ensembles quantify both aleatoric and epistemic uncertainty, and produce minimax optimal predictions under certain conditions.
Multi-objective Ensemble Generation

Shenkai Gu,Ran Cheng,Yaochu Jin

DOI: https://doi.org/10.1002/widm.1158

2015-01-01

Abstract:Ensemble methods that combine a committee of machine‐learning models, each known as a member or base learner, have gained research interests in the past decade. One interest on ensemble generation involves the multi‐objective approach, which attempts to generate both accurate and diverse members that fulfill the theoretical requirements of good ensembles. These methods resolve common difficulties of balancing the trade‐off between accuracy and diversity and have been shown to be advantageous over single‐objective methods. This study presents an up‐to‐date survey on multi‐objective ensemble generation methods, including widely used diversity measures, member generation, selection, and integration techniques. Challenges and potential applications of multi‐objective ensemble generation are also discussed. WIREs Data Mining Knowl Discov 2015, 5:234–245. doi: 10.1002/widm.1158This article is categorized under: Algorithmic Development > Ensemble Methods
Surrogate Maximization/minimization Algorithms and Extensions

Zhihua Zhang,James T. Kwok,Dit-Yan Yeung

DOI: https://doi.org/10.1007/s10994-007-5022-x

IF: 5.414

2007-01-01

Machine Learning

Abstract:Surrogate maximization (or minimization) (SM) algorithms are a family of algorithms that can be regarded as a generalization of expectation-maximization (EM) algorithms. An SM algorithm aims at turning an otherwise intractable maximization problem into a tractable one by iterating two steps. The S-step computes a tractable surrogate function to substitute the original objective function and the M-step seeks to maximize this surrogate function. Convexity plays a central role in the S-step. SM algorithms enjoy the same convergence properties as EM algorithms. There are mainly three approaches to the construction of surrogate functions, namely, by using Jensen’s inequality, first-order Taylor approximation, and the low quadratic bound principle. In this paper, we demonstrate the usefulness of SM algorithms by taking logistic regression models, AdaBoost and the log-linear model as examples. More specifically, by using different surrogate function construction methods, we devise several SM algorithms, including the standard SM, generalized SM, gradient SM, and quadratic SM algorithms, and their two variants called the conditional surrogate maximization (CSM) and surrogate conditional maximization (SCM) algorithms.
A Novel Surrogate-assisted Evolutionary Algorithm Applied to Partition-based Ensemble Learning

Arkadiy Dushatskiy,Tanja Alderliesten,Peter A. N. Bosman

DOI: https://doi.org/10.48550/arXiv.2104.08048

2021-04-16

Abstract:We propose a novel surrogate-assisted Evolutionary Algorithm for solving expensive combinatorial optimization problems. We integrate a surrogate model, which is used for fitness value estimation, into a state-of-the-art P3-like variant of the Gene-Pool Optimal Mixing Algorithm (GOMEA) and adapt the resulting algorithm for solving non-binary combinatorial problems. We test the proposed algorithm on an ensemble learning problem. Ensembling several models is a common Machine Learning technique to achieve better performance. We consider ensembles of several models trained on disjoint subsets of a dataset. Finding the best dataset partitioning is naturally a combinatorial non-binary optimization problem. Fitness function evaluations can be extremely expensive if complex models, such as Deep Neural Networks, are used as learners in an ensemble. Therefore, the number of fitness function evaluations is typically limited, necessitating expensive optimization techniques. In our experiments we use five classification datasets from the OpenML-CC18 benchmark and Support-vector Machines as learners in an ensemble. The proposed algorithm demonstrates better performance than alternative approaches, including Bayesian optimization algorithms. It manages to find better solutions using just several thousand fitness function evaluations for an ensemble learning problem with up to 500 variables.

Neural and Evolutionary Computing,Artificial Intelligence,Machine Learning
Structural reliability analysis based on ensemble learning of surrogate models

Kai Cheng,Zhenzhou Lu

DOI: https://doi.org/10.1016/j.strusafe.2019.101905

IF: 5.712

2020-03-01

Structural Safety

Abstract:Assessing the failure probability of complex structure is a difficult task in presence of various uncertainties. In this paper, a new adaptive approach is developed for reliability analysis by ensemble learning of multiple competitive surrogate models, including Kriging, polynomial chaos expansion and support vector regression. Ensemble of surrogates provides a more robust approximation of true performance function through a weighted average strategy, and it helps to identify regions with possible high prediction error. Starting from an initial experimental design, the ensemble model is iteratively updated by adding new sample points to regions with large prediction error as well as near the limit state through an active learning algorithm. The proposed method is validated with several benchmark examples, and the results show that the ensemble of multiple surrogate models is very efficient for estimating failure probability (>10−4) of complex system with less computational costs than the traditional single surrogate model.

engineering, civil

Ensembles of Surrogate Models

Support Vector Machines Ensemble With Optimizing Weights By Genetic Algorithm

Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

MID-ESM: An Adaptive Median-Based Ensemble of Surrogate Models

Deep Residual Surrogate Model

Ensemble of surrogates in black-box-type engineering optimization: Recent advances and applications

A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy

The Ensemble of Surrogate Model Based on Local and Global Errors

A Study on Metamodeling Techniques, Ensembles, and Multi-Surrogates in Evolutionary Computation

An Evolution Strategy Assisted by an Ensemble of Local Gaussian Process Models.

A surrogate-ensemble assisted expensive many-objective optimization

A Study on Metamodeling Techniques, Ensembles, and Multi-Surrogates in Surrogate-Assisted Memetic Algorithms

hubEnsembles: Ensembling Methods in R

Synergy-of-Experts: Collaborate to Improve Adversarial Robustness

An Ensemble Surrogate-based Framework for Expensive Multiobjective Evolutionary Optimization

Are Ensembles Getting Better all the Time?

Deep interpretable ensembles

Multi-objective Ensemble Generation

Surrogate Maximization/minimization Algorithms and Extensions

A Novel Surrogate-assisted Evolutionary Algorithm Applied to Partition-based Ensemble Learning

Structural reliability analysis based on ensemble learning of surrogate models